Comprehensive Azure Data Factory knowledge base with official documentation sources, CI/CD methods, deployment patterns, and troubleshooting resources
This skill inherits all available tools. When active, it can use any tool Claude has access to.
MANDATORY: Always Use Backslashes on Windows for File Paths
When using Edit or Write tools on Windows, you MUST use backslashes (\) in file paths, NOT forward slashes (/).
Examples:
D:/repos/project/file.tsxD:\repos\project\file.tsxThis applies to:
NEVER create new documentation files unless explicitly requested by the user.
This skill provides comprehensive reference information about Azure Data Factory, including official documentation sources, CI/CD deployment methods, and troubleshooting resources. Use this to access detailed ADF knowledge on-demand.
@microsoft/azure-data-factory-utilities requires Node.js 20.xBreaking Change:- Older Node.js versions (14.x, 16.x, 18.x) may cause package incompatibility errors- Update CI/CD pipelines to use Node.js 20.x or compatible versionsGitHub Actions:yaml- name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '20.x'Azure DevOps:yaml- task: UseNode@1 inputs: version: '20.x'Status: Available only for existing customers as of early 2025 Retirement Date: Not yet announced, but feature is officially deprecated Impact: New customers cannot provision Apache Airflow in Azure Data Factory
Official Deprecation Notice:
Migration Path:
Why Deprecated:
Action Required:
Reference:
ADF Mounting in Fabric:
Cross-Workspace Pipeline Orchestration:
Variable Libraries:
Connector Enhancements:
CRITICAL: As of 2025, npm package @microsoft/azure-data-factory-utilities requires Node.js 20.x
Breaking Change:
GitHub Actions:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20.x'
Azure DevOps:
- task: UseNode@1
inputs:
version: '20.x'
Main Documentation Hub:
Introduction to Azure Data Factory:
Library ID: /websites/learn_microsoft_en-us_azure_data-factory
How to Access:
Use Context7 MCP tool to fetch latest documentation:
mcp__context7__get-library-docs:
- context7CompatibleLibraryID: /websites/learn_microsoft_en-us_azure_data-factory
- topic: "CI/CD continuous integration deployment pipelines ARM templates"
- tokens: 8000
npm Package: @microsoft/azure-data-factory-utilities
Key Features:
package.json Configuration:
{
"scripts": {
"build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index",
"build-preview": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index --preview"
},
"dependencies": {
"@microsoft/azure-data-factory-utilities": "^1.0.3"
}
}
Commands:
# Validate resources
npm run build validate <rootFolder> <factoryId>
# Generate ARM templates
npm run build export <rootFolder> <factoryId> [outputFolder]
# Preview mode (only stop/start modified triggers)
npm run build-preview export <rootFolder> <factoryId> [outputFolder]
Official Documentation:
Method: Git integration + Publish button
Process:
adf_publish branchadf_publish branchWhen to Use:
Limitations:
Migration Path: Modern approach recommended for new implementations
Primary Command: New-AzResourceGroupDeployment
Syntax:
New-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>" `
-Mode Incremental `
-Verbose
Validation:
Test-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>"
What-If Analysis:
New-AzResourceGroupDeployment `
-ResourceGroupName "<resource-group-name>" `
-TemplateFile "ARMTemplateForFactory.json" `
-TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
-factoryName "<factory-name>" `
-WhatIf
Primary Command: az deployment group create
Syntax:
az deployment group create \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name> \
--mode Incremental
Validation:
az deployment group validate \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name>
What-If Analysis:
az deployment group what-if \
--resource-group <resource-group-name> \
--template-file ARMTemplateForFactory.json \
--parameters ARMTemplateParametersForFactory.<environment>.json \
--parameters factoryName=<factory-name>
Key Improvement in Ver2:
Download Command:
# Linux/macOS/Git Bash
curl -o PrePostDeploymentScript.Ver2.ps1 https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1
# PowerShell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1" -OutFile "PrePostDeploymentScript.Ver2.ps1"
Pre-Deployment (Stop Triggers):
./PrePostDeploymentScript.Ver2.ps1 `
-armTemplate "<path-to-ARMTemplateForFactory.json>" `
-ResourceGroupName "<resource-group-name>" `
-DataFactoryName "<factory-name>" `
-predeployment $true `
-deleteDeployment $false
Post-Deployment (Start Triggers & Cleanup):
./PrePostDeploymentScript.Ver2.ps1 `
-armTemplate "<path-to-ARMTemplateForFactory.json>" `
-ResourceGroupName "<resource-group-name>" `
-DataFactoryName "<factory-name>" `
-predeployment $false `
-deleteDeployment $true
Version: PowerShell Core (7.0+) recommended
pwsh: true in AzurePowerShell@5 taskpwsh command, not powershellModules Required:
Official Documentation:
Medium Article (Recent 2025):
Microsoft Community Hub:
Community Blog (February 2025):
Essential Actions:
actions/checkout@v4 - Checkout repositoryactions/setup-node@v4 - Setup Node.jsactions/upload-artifact@v4 - Publish ARM templatesactions/download-artifact@v4 - Download ARM templates in deploy workflowazure/login@v2 - Authenticate to Azureazure/arm-deploy@v2 - Deploy ARM templatesazure/powershell@v2 - Run PrePostDeploymentScriptService Principal (JSON credentials):
{
"clientId": "<GUID>",
"clientSecret": "<STRING>",
"subscriptionId": "<GUID>",
"tenantId": "<GUID>"
}
Store in GitHub secret: AZURE_CREDENTIALS
Workload Identity Federation (More secure):
Microsoft Learn:
Community Guides:
Towards Data Science:
Build Pipeline Tasks:
UseNode@1 - Install Node.jsNpm@1 - Install packages, run build commandsPublishPipelineArtifact@1 - Publish ARM templatesRelease Pipeline Tasks:
DownloadPipelineArtifact@2 - Download ARM templatesAzurePowerShell@5 - Run PrePostDeploymentScriptAzureResourceManagerTemplateDeployment@3 - Deploy ARM templatePermissions Needed:
Configuration:
URL: https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide Last Updated: January 2025
Common Issues Covered:
Enable Diagnostic Settings:
Azure Portal → Data Factory → Diagnostic settings → Add diagnostic setting
Send to: Log Analytics workspace
Logs to Enable:
- PipelineRuns
- TriggerRuns
- ActivityRuns
- SandboxPipelineRuns
- SandboxActivityRuns
Kusto Queries for Troubleshooting:
// Failed pipeline runs in last 24 hours
ADFPipelineRun
| where Status == "Failed"
| where TimeGenerated > ago(24h)
| project TimeGenerated, PipelineName, RunId, Status, ErrorMessage, Parameters
| order by TimeGenerated desc
// Failed CI/CD deployments
ADFActivityRun
| where ActivityType == "ExecutePipeline"
| where Status == "Failed"
| where TimeGenerated > ago(7d)
| project TimeGenerated, PipelineName, ActivityName, ErrorCode, ErrorMessage
| order by TimeGenerated desc
// Performance analysis
ADFActivityRun
| where TimeGenerated > ago(7d)
| extend DurationMinutes = datetime_diff('minute', End, Start)
| summarize AvgDuration = avg(DurationMinutes) by ActivityType, ActivityName
| where AvgDuration > 10
| order by AvgDuration desc
Error: "Template parameters are not valid"
Error: "Updating property type is not supported"
Error: "Operation timed out"
Error: "Authentication failed"
Recommended Folder Layout:
repository-root/
├── adf-resources/ # ADF JSON files (if using npm approach)
│ ├── dataset/
│ ├── pipeline/
│ ├── trigger/
│ ├── linkedService/
│ └── integrationRuntime/
├── .github/
│ └── workflows/ # GitHub Actions workflows
│ ├── adf-build.yml
│ └── adf-deploy.yml
├── azure-pipelines/ # Azure DevOps pipelines
│ ├── build.yml
│ └── release.yml
├── parameters/ # Environment-specific parameters
│ ├── ARMTemplateParametersForFactory.dev.json
│ ├── ARMTemplateParametersForFactory.test.json
│ └── ARMTemplateParametersForFactory.prod.json
├── package.json # npm configuration
└── README.md
Only Configure Git on Development ADF:
Rationale: Prevents accidental manual changes in higher environments
Environment Flow:
Dev (Git) → Build → Test → Approval → Production
↓
ARM Templates
Parameter Management:
Set up alerts for:
Recommended Tools:
Official Azure Data Factory Samples:
Community Examples:
Microsoft Q&A:
Stack Overflow:
azure-data-factoryAzure Status:
Situations requiring current documentation:
How to Fetch:
This knowledge base should be your starting point for all Azure Data Factory questions. Always verify critical information with the latest official documentation when making production decisions.