Skip to main content

Data upload for Unix-based systems

Updated over a year ago

Standard setup in a Unix shell:

In the following, we provide an upload script and an explanation of how to schedule it. Make sure that the necessary requirements of your system are met for execution.

Step 1: Prerequisites

Please install the following programs listed in the table and add them to your system PATH accordingly. A detailed description of this process can be found below.

Tool

Version

Command

Azure CLI

>=2.53.0

az version

AzCopy

>=10.20.1

azcopy --version

Adding system environment variables to the PATH on UNIX devices

In this guide, we will walk you through the process of adding a user-defined directory to the system's PATH environment variable. We describe the procedure for the program Azcopy, which we want to add to our PATH.

The PATH is an environment variable and contains a list of folders / directories in which the system searches for executable programs. In order for any program to be executed via the terminal, the executable file must be stored in one of the folders listed in the PATH

  1. Check: Is the program already recognized in the terminal?

    which azcopy


    If this is not the case, the following information is provided:

    azcopy not found

  2. Identify the directory: Specify the path to the directory you want to add to the PATH and replace /Path/to/azcopy-Verzeichnis in the following code examples with the actual path.
    ​

  3. Edit the user's shell configuration file: Open the corresponding terminal configuration file. Common files are .bashrc, .zshrc or .profile. Use your preferred text editor to open the file.
    ​
    In this example, we open the configuration file using Nano:
    ​

    nano ~/.bashrc

  4. Add the directory to the PATH: Add the following code at the end of the file and replace /path/to/azcopy directory as mentioned before. This will add the directory to the PATH.
    ​

    export PATH=/Path/to/azcopy-Verzeichnis:$PATH

  5. Save and exit: Save the file and exit the text editor.

  6. Apply changes: Either open a new terminal session or apply the changes immediately to your current session with the following command:

    source ~/.bashrc

  7. Checking the process: You will see the updated PATH with the added directory, which is now permanently available. The path to our azcopy program should now also be in the first position.

    echo $PATH

    When executing the following command again, azcopy should now be recognized and the installation path should be displayed.

    which azcopy

    To check whether the tools have been installed correctly and added to the path, open a terminal and enter the commands listed in the table.

    az version
    azcopy --version

Step 2: Setting up the data transfer:

On Unix systems, the upload can be carried out using a shell script. You will receive the script prepared for you with your access data from the contact person responsible for your onboarding.

You can then carry out the following steps:

Set up folder structure

  1. Create a “tacto” folder on your local computer from which you transfer the data exports to Tacto.

  2. You must then create 3 more folders within the “tacto” folder:

    1. "data-export"

    2. "logs"

    3. "script"

  3. Place your first bundle of export files in the “data-export” folder.

  4. Also create a .log file within the “logs” folder with the name “scheduled-task.log”. In the future, log data will be saved in this file during data transfer.

  5. Adjust the corresponding placeholders in the script as follows:

SOURCE_FOLDER ="Path/to/your/local/data-export-folder" LOGPATH="Path/to/your/log-file/scheduled-task.log"

  1. Save the script in a local directory on your machine.
    ​

  2. Give the .sh script execution rights. It is important that you are in the folder in which the script is located in order to execute the command

chmod +x <fileName>


Step 3: Planning the automated data transfer with cron tasks

The execution of the script can be automated via cron tasks. The following steps are necessary for this:

  1. Open a terminal window and enter the command “crontab -e” to open the crontab file.

  2. If this is the first time you are using Cron, a text editor such as nano or vim may open where you can add the tasks. Select the editor you want to work with.

  3. Enter the following line at the end of the crontab file to execute the script “data-update-script.sh” weekly:

    0 0 * * 0 /path/to/your/folder/data-update-script.sh -tee

    This command executes the script every Sunday at midnight. Note that you need to adjust the path to the script to specify the actual directory where the script is located.

    To run the script at a different time, adjust the schedule code. A cheat sheet for the corresponding codes can be found here.

  4. Save the file and close the editor.

Now the script “data-update-script.sh” is executed at the desired frequency as long as the cron daemon is running. You can check the cron daemon by entering the command systemctl status cron in the terminal. If Cron is not active, you can start it with the command “systemctl start cron”.
​

🏁 You have now successfully set up the automatic data upload to the Tacto cloud!

Detailed explanation of the Azure CLI tool and its use in connection with Tacto

❗The following section describes the more detailed use of the Azure CLI tool. The scope of this information goes far beyond the standard case.

For the interaction between your systems and the Tacto Azure Blob Storage, mainly az storage fs commands of the Azure CLI tool are used. The following summaries are intended to provide an overview of the most important functions. For further information, please refer to the documentation (Docs).

❗There is currently a problem with the Azure CLI command for batch uploads as a service principal (see GitHub issue).

As a workaround, it is recommended to use the azcopy utility directly instead of the integrated azcopy (Docs).

Uploading a single file

  1. Log in with the service principal provided (see credentials in the e-mail).

    APP_ID=<APP_ID> 
    CLIENT_SECRET=<CLIENT_SECRET> 
    TENANT_ID=<TENANT_ID> 
    FOLDER_NAME=<FOLDER_NAME> 

    az login --allow-no-subscriptions --service-principal \
    -u ${APP_ID} -p ${CLIENT_SECRET} -t ${TENANT_ID}

  2. Upload file.

    az storage fs file upload --auth-mode login \
    --file-system erp-data --account-name tactoexchange \
    --path ${FOLDER_NAME}/<FILE_NAME> --source <SOURCE_PATH>

Batch upload of files

  1. Log in with the service principal provided (see credentials in the e-mail).

    # ---SETTINGS---
    # SOURCE_FOLDER needs to be replaced with the customer's directory
    APP_ID=<APP_ID> 
    CLIENT_SECRET=<CLIENT_SECRET> 
    TENANT_ID=<TENANT_ID> 
    DESTINATION_FOLDER=<customer-id>
    SOURCE_FOLDER=<"$HOME/tacto/data-export">

    # ---only modify the settings above---

    export AZCOPY_SPA_CLIENT_SECRET=${CLIENT_SECRET} 
    azcopy login --service-principal --application-id ${APP_ID} \
    --tenant-id ${TENANT_ID}

  2. Upload file.

    azcopy copy "${SOURCE_FOLDER}/*" "https://tactoexchange.dfs.core.windows.net/erp-data/${DESTINATION_FOLDER}" --recursive --overwrite=False

    ℹ️ According to the documentation, the source and destination paths should be embedded in single quotation marks (').

Read files

  1. Log in with the service principal provided (see credentials in the mail).

    APP_ID=<APP_ID> 
    CLIENT_SECRET=<CLIENT_SECRET> 
    TENANT_ID=<TENANT_ID> 
    DESTINATION_FOLDER=<customer-id>

    az login --allow-no-subscriptions --service-principal -u ${APP_ID} -p ${CLIENT_SECRET} -t ${TENANT_ID}

  2. Output all files in the folder.

    az storage fs file list --file-system erp-data --account-name tactoexchange \ 
    --output table --auth-mode login \ 
    --path ${FOLDER_NAME}
Related Articles
Set up automatic data upload
Data upload for Windows-based systems
Data upload with Lobster (SFTP)
Did this answer your question?