You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Sep 25, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: docs/how-tos/airflow/use-aws-secrets-manager.md
+79-11Lines changed: 79 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,26 +22,29 @@ Each time a variable is accessed, an API call is made to AWS Secrets Manager. If
22
22
23
23
### To solve for this there are 2 best practices to follow:
24
24
25
-
1. Always call your `Variable.get` from within the Datacoves Task Decorator. This ensures the variable is only fetched at runtime.
26
-
2. Make use of the `connections_lookup_pattern` and `variables_lookup_pattern` when setting up your secondary backend above. This means only variables and connections prefixed with `aws_` would be make an API call to AWS Secrets Manager. eg) `aws_mayras_secret`
25
+
1. Always call your `Variable.get` from within an Airflow decorator such as the Datacoves Bash Task Decorator. This ensures the variable is only fetched at run time.
26
+
2. Make use of the `connections_lookup_pattern` and `variables_lookup_pattern` when setting up your secondary backend above. This means only variables and connections prefixed with `aws_` would be make an API call to AWS Secrets Manager. eg) `aws_my_secret`
27
27
28
28
29
29
```python
30
+
"""
31
+
## Sample DAG using variables
32
+
This DAG is a sample using the Datacoves decorators with variable from AWS.
33
+
"""
34
+
30
35
from airflow.decorators import dag, task
31
36
from pendulum import datetime
32
37
from airflow.models import Variable
33
38
34
-
doc ="""## Datacoves Bash Decorator DAG
35
-
This DAG is a sample using the Datacoves decorators with variable calls."""
my_var = Variable.get("aws_mayras_secret") # Call variable within @task.datacoves_bash
56
+
my_var = Variable.get("aws_my_secret") # Call variable within @task.datacoves_bash
54
57
returnf"My variable is: {my_var}"
55
58
56
59
calling_vars_in_decorator() # Call task function
57
60
58
61
# Invoke Dag
59
-
dag = task_decorators_example()
60
-
dag.doc_md = doc
62
+
task_decorators_example()
63
+
```
64
+
65
+
>[!TIP]To auto mask your secret you can use `secret` or `password` in the secret name since this will set `hide_sensitive_var_conn_fields` to True. eg) aws_my_password. Please see [this documentation](https://www.astronomer.io/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables) for a full list of masking words.
66
+
67
+
## Using a secrets manager directly from Airflow
68
+
69
+
While not recommended, you can bypass the Datacoves secrets manager integration by configuring an Airflow connection and using the `SecretsManagerHook` in an Airflow DAG.
70
+
71
+
### Configure an Airflow Connection
72
+
Create a new Airflow Connection with the following parameters:
73
+
74
+
Connection Id: aws_secrets_manager
75
+
Connection Type: Amazon Web Services
76
+
AWS Access Key ID: ....
77
+
AWS Secret Access Key: ....
78
+
Extra:
79
+
{
80
+
"region_name": "us-west-2"
81
+
}
61
82
83
+
84
+
```python
85
+
"""
86
+
## Sample DAG using variables
87
+
This DAG is a sample using the Datacoves decorators with variable from AWS.
88
+
"""
89
+
90
+
from airflow.decorators import dag, task
91
+
from pendulum import datetime
92
+
from airflow.providers.amazon.aws.hooks.secrets_manager import SecretsManagerHook
var = secrets_manager_hook.get_secret("airflow/variables/aws_ngtest")
114
+
returnf"export MY_VAR={var} && echo $MY_VAR"
115
+
116
+
aws_var()
117
+
118
+
variable_usage()
62
119
```
63
120
64
-
>[!TIP]To auto mask your secret you can use `secret` or `password` in the secret name since this will set `hide_sensitive_var_conn_fields` to True. eg) aws_mayras_password. Please see [this documentation](https://www.astronomer.io/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables) for a full list of masking words.
121
+
## Check when secret is being fetched from AWS
122
+
123
+
It is a good idea to verify that Secrets are only being fetched when expected. To do this, you can use AWS CloudTrail.
124
+
125
+
1. From the AWS Console, go to `CloudTrail`
126
+
2. Click `Event History`
127
+
3. Click `Clear Filter`
128
+
4. In the `Lookup Attributes` dropdown, select `Event Name`
129
+
5. In the `Enter an Event Name` input box, enter `GetSecretValue`
130
+
131
+
Review the `Resource name` and `Event time`.
132
+
Note: it may take a few minutes for fetch events to show up in CloudTrail.
Copy file name to clipboardExpand all lines: docs/how-tos/vscode/reset-user-env.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
1
# Reset the User's Env if the git Repository is Changed in the Project.
2
2
3
-
If you change the repo associated with your environment after one has already been cloned into it then you will need to remove the cloned repo in the transform and reset the environment.
3
+
If you need to reset your user environment because you change the repo associated with your environment after one has already been cloned or if the repo fails to clone, you will need to remove the workspace folder and reset the environment.
4
4
5
-
- Open terminal and enter the following commands.
5
+
- Open terminal and enter the following commands. If you get an error when opening the terminal because you don't have a `transform` folder or similar, simply right click in the file area and select `Open in Integrated Terminal`
0 commit comments