DPDP Act Requirements for AI Software in India

DPDP Act Requirements for AI Software in India

The DPDP Act requirements for AI software in India focus on lawful processing of personal data, clear notice and consent where required, security safeguards, user rights, breach response, and responsible use of processors. The Act does not prohibit AI software, but it requires organisations to understand what personal data their AI system uses, why it uses it, where it goes, and how individuals can exercise their rights.

An AI application used by an Indian school, clinic, D2C brand, NGO, agency or small business may fall within the Digital Personal Data Protection Act, 2023 if it processes digital personal data about individuals in India. The exact compliance work depends on the type of data, the AI system’s purpose, the organisation’s role, the number and sensitivity of affected users, and any sector-specific rules.

This article explains the practical requirements for building, buying or integrating AI software in India. It is not a substitute for advice from a qualified legal or privacy professional. The status of supporting rules and sector guidance should be checked before launch because implementation requirements may be updated or notified in stages.

When the DPDP Act Applies to AI Software

The DPDP Act applies to the processing of digital personal data. “Processing” is broad. It includes collecting, storing, organising, using, sharing, analysing, altering and deleting data.

For an AI system, processing may happen in more places than the visible application. Consider a customer-support chatbot. It may process:

• Names and mobile numbers entered by customers
• Chat messages and complaint details
• Order information
• Voice recordings or uploaded images
• Device and usage information
• Data sent to an external AI model provider
• Logs retained for monitoring or model improvement
• Human review records created to assess AI responses

If this information identifies, or can reasonably identify, an individual, it may be personal data. The organisation using the system may have DPDP obligations even if the AI model itself is supplied by another company.

Indian and overseas processing

The Act can apply when personal data is processed digitally in India. It can also apply to processing outside India when the processing relates to offering goods or services to individuals in India.

This matters for AI tools hosted outside India. A company cannot assume that a foreign cloud region or overseas AI API removes Indian privacy responsibilities. An Indian business using a US, European or Asian AI service still needs to assess the data flow, contract terms, security controls and transfer arrangements.

The DPDP Act allows the Central Government to restrict transfers of personal data to specified countries or territories. Organisations should therefore avoid treating cross-border transfer as permanently risk-free. They should check current notifications and their sector regulator’s requirements.

Who is responsible?

The organisation deciding why and how personal data is processed is generally the Data Fiduciary. In practical terms, this could be:

• A clinic deciding to use an AI assistant for appointment triage
• A school deploying an AI learning platform
• A D2C brand using AI for customer segmentation
• An NGO collecting beneficiary data for an AI-based support tool
• An agency building a chatbot for its client

An AI vendor or cloud provider processing data on the organisation’s instructions may be a Data Processor. The organisation remains responsible for selecting, instructing and monitoring that processor.

A contract alone does not remove the Data Fiduciary’s responsibilities. The business must know what the vendor does with the data, whether it uses data to train shared models, how long it retains prompts and logs, who can access them, and how data is deleted.

Identify Personal Data in the AI Workflow

Before selecting an AI model or writing a privacy policy, map the full data lifecycle. This is one of the most important DPDP Act requirements for AI software in India because AI products often send data through multiple services.

Create an inventory covering:

1. Data collection – What does the user enter, upload, record or generate?
2. Data storage – Where are prompts, files, embeddings, transcripts and logs stored?
3. Model processing – Which model receives the data, and for what purpose?
4. Human review – Can staff, contractors or vendor reviewers see the content?
5. Training and improvement – Is data retained for model training, evaluation or fine-tuning?
6. Sharing – Is data shared with cloud providers, analytics tools, CRM systems or other vendors?
7. Retention – When are source files, prompts, outputs and backups deleted?
8. User rights – Can the organisation find, correct, export or delete a person’s data?

AI data that businesses often overlook

AI compliance is not limited to the information used to train a model. The following may also be personal data:

• Prompt history
• Chat transcripts
• Uploaded resumes
• Customer support tickets
• Voice recordings
• Facial or identity documents
• Medical descriptions
• Student performance records
• Employee feedback
• IP addresses and account identifiers
• Vector embeddings linked to a person
• Model evaluation datasets
• Moderation records
• Error reports containing real user content

A business should not label data as anonymous merely because names have been removed. If a person can still be identified by combining the dataset with other information, the data may remain personal data.

Use synthetic or de-identified data where practical

For testing, demonstrations and development, synthetic data can reduce privacy risk. For example, a school building a question-answering assistant can test the interface with fictional student records rather than actual marksheets.

De-identification may also help, but it requires care. Remove direct identifiers, reduce unnecessary attributes, control access to the re-identification key and document the method. If the data can be linked back to individuals, treat it as personal data for compliance planning.

Notice, Consent and Lawful Purpose

The DPDP Act requires processing to have a lawful purpose. In many AI applications, this will involve consent, although the Act also recognises certain legitimate uses set out in the law.

Consent should be:

• Free
• Specific
• Informed
• Unconditional
• Unambiguous
• Given through clear affirmative action

A vague statement such as “we may use your information to improve services” may not adequately explain an AI workflow. A practical notice should tell the person what data is collected, why it is used, whether it is sent to an AI vendor, how long it is retained, and how rights can be exercised.

Consent for an AI chatbot

Suppose a clinic uses a chatbot to collect symptoms before an appointment. The notice should not hide the AI purpose in a long document. It should explain:

• That the person is interacting with an AI system, if relevant to the experience
• What information the system asks for
• Whether the information will be reviewed by clinic staff
• Whether an external model provider processes the conversation
• Whether the information is used only for the appointment or also for service improvement
• How to request correction or deletion
• How to contact the clinic about privacy concerns

The consent mechanism should be separate from unrelated marketing consent where the purposes are different. A customer should not be forced to agree to AI model training or promotional communication merely to place an order, unless the business has a legally supportable reason and clear user choice.

Withdrawal of consent

The Act provides a right to withdraw consent. Withdrawal should be as easy as giving consent. If consent was collected through a website checkbox, the business should provide a practical method through the account area, email, customer-support channel or another suitable route.

Withdrawal does not necessarily mean that every historical processing activity can be reversed immediately. However, the organisation should stop the relevant processing unless another lawful basis applies, and it should explain any retention required by law or a legitimate operational need recognised under the Act.

Purpose limitation for AI

AI systems tend to expand beyond their original use. A company may begin with an internal document assistant and later consider using the same content for employee scoring, advertising or model training. These are different purposes and should be assessed separately.

Do not collect broad data “for future AI use” without a defined purpose. Write down the original business purpose, the categories of data required and the permitted uses. If the use changes materially, review the notice, consent and legal basis before making the change.

Data Fiduciary and Vendor Responsibilities

The business deploying AI software is usually responsible for the compliance outcome. It should conduct vendor due diligence before connecting personal data to a model, SaaS platform or API.

Ask the vendor:

• Does the service use customer prompts or files to train a general model?
• Can training be disabled by contract or account setting?
• Where are data, logs and backups stored?
• How long are prompts and outputs retained?
• Are subcontractors involved?
• Can the vendor delete data on instruction?
• What access controls and encryption are used?
• How are security incidents reported?
• Can the organisation retrieve data needed for a rights request?
• Does the vendor support audit evidence?
• What happens to data after contract termination?

The answers should be documented, not left only in a sales call. Review the privacy policy, data-processing terms, security documentation and product configuration.

Contract clauses for AI vendors

A contract should define the vendor’s role and instructions. It should address:

• Permitted processing purposes
• Categories of personal data
• Security measures
• Confidentiality obligations
• Sub-processors
• Breach notification
• Assistance with access, correction and erasure requests
• Retention and deletion
• Data location and transfer
• Audit or assurance rights
• Return or deletion when the service ends
• Restrictions on independent model training

A vendor promising that it “does not sell data” may still retain prompts, use them for product improvement or share them with subprocessors. Selling is only one possible privacy concern.

Rights of Individuals Using AI Systems

The DPDP Act provides rights for Data Principals, meaning the individuals to whom personal data relates. AI software must be designed so the organisation can respond to these rights in a controlled way.

Access to information

Individuals may request information about their personal data and related processing. The organisation should be able to locate data across the application database, support platform, file storage, AI provider and relevant logs.

This is difficult when an AI tool stores information in conversation history or vector databases without a reliable user identifier. Build an identity link at the time of collection, while avoiding unnecessary collection of extra identifiers.

Correction and erasure

Individuals may request correction of inaccurate or incomplete personal data and erasure where applicable. AI systems create a practical complication: changing a database record may not automatically remove the same information from cached prompts, embeddings, exported files or evaluation datasets.

A reasonable design should identify all locations in which personal data is copied. It should also define what the business can delete from the model provider’s systems and what must be removed from the organisation’s own systems.

Do not promise that a generative AI model can simply “forget” a specific item unless the technical and contractual process has been verified. If the vendor cannot support deletion from training data, do not send unnecessary personal data for training or fine-tuning.

Grievance redressal

The organisation should publish a contact method for privacy grievances and assign ownership internally. A small business may use a monitored privacy email address and a documented escalation process. A larger organisation may need a dedicated privacy or compliance function.

Support staff should know how to recognise a data request. A message saying “delete all my chats” should not be treated as an ordinary technical ticket and then lost in the queue.

Nomination

The Act includes a right to nominate another individual to exercise rights in the event of death or incapacity, subject to the applicable process. Systems handling long-term member, patient, donor or account information should consider how such requests will be verified and recorded.

Security Safeguards and Data Breach Response

The Data Fiduciary must take reasonable security safeguards to prevent personal data breaches. For AI systems, this requires more than protecting the main application database.

Important controls include:

• Encryption in transit and at rest
• Strong administrator authentication
• Role-based access for staff and vendors
• Separation of development, testing and production data
• Secure API key storage
• Prompt and file access controls
• Audit logs
• Rate limiting and abuse monitoring
• Backup protection
• Vulnerability management
• Vendor access reviews
• Secure deletion
• Staff training on confidential prompts

AI applications also introduce model-specific risks. A user may try to extract hidden instructions, confidential documents or another person’s conversation. Retrieval-augmented systems may expose documents when permissions are configured incorrectly. These are privacy issues as well as application-security issues.

Breach response

A breach could involve:

• A leaked API key
• Public access to a bucket containing prompts
• An AI vendor compromise
• Accidental disclosure in a generated answer
• A staff member uploading sensitive records to an unauthorised tool
• Cross-tenant access to another customer’s documents
• Malware encrypting AI application data

Maintain an incident response plan with named owners, vendor contacts, investigation steps, evidence preservation and communication procedures. The DPDP Act includes obligations regarding notification of personal data breaches to the Board and affected Data Principals, subject to the applicable requirements and procedures. Organisations should monitor notified rules and official guidance for the required format and timing.

Do not wait until an incident occurs to decide whether the AI provider must notify you. Put an escalation period and contact process in the contract.

Children, Sensitive Use Cases and Sector Rules

The DPDP Act has specific provisions for children and applies additional obligations to processing children’s personal data. It defines a child as an individual who has not completed eighteen years of age.

AI software used by schools, coaching classes, child-focused platforms and family services should therefore include age-related controls. These may include:

• Age declaration or verification where appropriate
• Verifiable parental consent mechanisms
• Restrictions on behavioural monitoring
• Controls against targeted advertising to children
• Review of nudging, profiling and engagement features
• Human oversight for high-impact recommendations

A school should not assume that a general-purpose AI tool is suitable for student data merely because it has an education-focused interface. Student records, learning difficulties, family information and disciplinary records require careful access control and retention.

Health, finance and identity data

The DPDP Act should be considered alongside sector obligations. A clinic may need to assess health-record confidentiality, professional duties and applicable health-data guidance. A fintech or lender must consider RBI requirements, outsourcing rules and customer-data controls. Insurance, securities, telecom, education and government-linked projects may have their own requirements.

Aadhaar data, payment data, employee records and health information can create additional contractual, regulatory and security expectations even where the DPDP Act does not use a separate category for every type of sensitive data.

Do not use an AI tool for medical diagnosis, credit decisions, recruitment or student evaluation without reviewing the applicable sector and professional requirements. The DPDP Act is not the only compliance framework that may apply.

Significant Data Fiduciaries and Higher-Risk AI

The Government may notify certain organisations as Significant Data Fiduciaries based on factors such as the volume and sensitivity of personal data, risk to India’s sovereignty and integrity, risk to electoral democracy, security of the State, public order and other relevant considerations.

A Significant Data Fiduciary can face additional obligations, including appointing a Data Protection Officer based in India, appointing an independent data auditor, conducting impact assessments and audits, and meeting other requirements prescribed under the law.

Most small businesses will not automatically fall into this category. However, the risk assessment is still useful. An NGO managing sensitive beneficiary data, a health platform serving a large population or a technology provider supporting public-facing systems should not wait for a formal designation before adopting stronger controls.

AI impact assessment

Even where a formal impact assessment is not legally required, conduct a practical review for AI systems that affect people significantly. Ask:

• Could the output deny a service, job, loan, admission or benefit?
• Is the training data representative and accurate?
• Can a person challenge an incorrect result?
• Is there human review?
• Are proxy variables creating unfair outcomes?
• Are prompts and outputs logged safely?
• Can the organisation explain the system’s purpose and limits?
• Is the model appropriate for the language and context in which it is used?

The DPDP Act does not create a general EU-style right to an explanation for every automated decision, nor does it impose a blanket ban on automated decision-making. Nevertheless, clear internal review and human escalation are sensible controls for high-impact applications.

AI Compliance Checklist for Indian Businesses

Use this checklist before launching an AI feature or connecting an existing product to an AI service.

This checklist should be adapted to the organisation’s size and risk. A local agency building an internal document assistant may need a simpler process than a health platform processing thousands of patient interactions, but both should understand their data flows.

Common Mistakes to Avoid

Uploading customer data into free AI tools

Staff may paste customer complaints, employee issues or patient information into a public AI chatbot to draft a response. This can create an unauthorised disclosure and may bypass the organisation’s approved vendor controls.

Create a written acceptable-use policy. Provide an approved tool, restrict sensitive information, and train staff to anonymise content before using AI for drafting or analysis.

Assuming the AI vendor is the only responsible party

The organisation that decides to use the AI system generally remains accountable for its processing decisions. Vendor due diligence and a contract are necessary, but they do not replace internal governance.

Using one consent for unrelated purposes

Consent to use a chatbot for customer support is not automatically consent to use conversations for advertising, employee profiling or model training. Separate purposes should be assessed and communicated clearly.

Keeping everything forever

Prompt logs can contain more information than the original form. Retaining them indefinitely increases the impact of a breach and makes rights requests harder. Define a retention period based on the purpose, legal requirements and operational need.

Treating AI output as verified fact

Privacy compliance does not make an AI answer accurate. A wrong output can expose another person’s information, misstate a patient’s condition or incorrectly classify a customer. Use access controls, output review and human escalation where the consequences matter.

Frequently Asked Questions

Does every AI tool used by an Indian business need DPDP compliance?

Not every AI tool has the same level of risk, but the DPDP Act may apply whenever the tool processes digital personal data connected to individuals in India. An internal AI tool handling only genuinely anonymous information may have limited DPDP exposure, while a chatbot using customer names, conversations or account details requires a proper assessment.

Is consent always required before using personal data with AI?

No. The DPDP Act recognises consent as a major basis for processing and also provides for certain legitimate uses. Whether consent is required depends on the purpose, the type of processing and the applicable provisions, so the organisation should document its reasoning rather than assume that one answer applies to every feature.

Can an Indian company use OpenAI, Google, Microsoft or another overseas AI service?

Potentially, but the company must assess cross-border processing, vendor terms, retention, security, subprocessors and any current restrictions or sector requirements. Do not send personal data to an overseas service until you know whether prompts are used for training, how deletion works and what contractual protections are available.

Does the DPDP Act require AI software to be explainable?

The Act does not establish a universal, EU-style explainability requirement for every AI output. However, organisations still need clear purposes, notices, rights processes and suitable safeguards, and high-impact systems should include human review and a way to challenge or correct harmful results.

What should a small business do if it cannot afford a full privacy team?

Start with a data inventory, approved-tool policy, clear privacy notice, vendor review, access controls, retention schedule and a basic rights and breach-response process. Assign an internal owner and obtain professional advice for high-risk use cases such as health, finance, children’s data or large-scale profiling.

Where to Start

Before buying or building AI software, list every category of personal data the system will receive and every provider that will handle it. Confirm the purpose, review whether consent or another lawful basis applies, and test the vendor’s retention, training, deletion and breach-notification terms.

Then create a small pilot using limited or synthetic data. Test access controls, user notices, deletion, correction requests and incident escalation before connecting production records. Review the result with legal, technical and operational stakeholders, especially when the AI system affects children, patients, employees, donors, customers or applicants.

For implementation choices, security controls and a privacy-aware AI workflow, talk to the Govindani Infotech team on WhatsApp; project scope and pricing are confirmed there.