Skip to main content
This guide explains how to access Google Cloud Storage (GCS) buckets from on-premise machines using filesystem mounting or command-line access.
Use Method 1 (Filesystem Mounting) for seamless integration with Visual Layer. This approach allows you to work with your data as if it were stored locally.

​
Prerequisites

Before you begin, ensure you have:
  • Linux-based system (Ubuntu/Debian recommended).
  • Internet connectivity.
  • Google Cloud account with access to the target bucket.
  • Appropriate permissions on the GCS bucket.

​
Method 1: Filesystem Mounting with GCS FUSE

This method provides direct access to your GCS bucket without copying files locally. GCS FUSE mounts a bucket as a local filesystem, making it appear as a regular folder on your machine.

​
Install Google Cloud SDK

Update your system and install required dependencies: 
sudo apt-get update && sudo apt-get install -y curl unzip python3
Download and install the Google Cloud CLI: 
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-465.0.0-linux-x86_64.tar.gz
tar -xf google-cloud-cli-465.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/install.sh
Restart your shell to apply the changes: 
exec -l $SHELL

​
Authenticate with Google Cloud

Choose one of the following authentication methods based on your use case.

​
User Authentication

Authenticate using your Google account: 
gcloud auth login user@example.com
This command opens a browser window (or provides a link to copy) where you sign in with your Google account and generate an authentication code to paste back into the terminal.

​
Service Account Authentication

For automated processes or production environments, use a service account. Create and download a service account key file from the Google Cloud Console, then set the environment variable: 
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"
Authenticate using the service account: 
gcloud auth activate-service-account --key-file="/path/to/your/service-account-key.json"

​
Install GCS FUSE

Add the Google Cloud packages repository: 
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/cloud.google.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt gcsfuse-jammy main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
Update package list and install GCS FUSE: 
sudo apt-get update
sudo apt-get install -y gcsfuse

​
Create Mount Point

Create a directory where you want to mount the bucket: 
mkdir -p ~/gcs_mount

​
Authenticate for GCS FUSE

Choose the authentication method that matches what you used when authenticating with Google Cloud. For User Authentication: 
gcloud auth application-default login
For Service Account Authentication: If you’re using a service account, the GOOGLE_APPLICATION_CREDENTIALS environment variable will be automatically used by GCS FUSE.

​
Mount the Bucket

Mount your GCS bucket to the local filesystem: 
gcsfuse your-bucket-name ~/gcs_mount
Example: 
gcsfuse my-example-bucket ~/gcs_mount

​
Verify the Mount

Verify that the bucket is successfully mounted: 
ls ~/gcs_mount
You should see the contents of your bucket.

​
Method 2: Command-Line Access

Use this method when you need to copy specific files locally before running Visual Layer profiler, or for one-time data transfers.

​
Install Google Cloud SDK

If you haven’t already installed it for Method 1, update your system and install required dependencies: 
sudo apt-get update && sudo apt-get install -y curl unzip python3
Download and install the Google Cloud CLI: 
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-465.0.0-linux-x86_64.tar.gz
tar -xf google-cloud-cli-465.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/install.sh
Restart your shell to apply the changes: 
exec -l $SHELL

​
Authenticate with Google Cloud

Choose one of the following authentication methods based on your use case.

​
User Authentication

Authenticate using your Google account: 
gcloud auth login user@example.com

​
Service Account Authentication

For automated processes or production environments, use a service account. Create and download a service account key file from the Google Cloud Console, then set the environment variable: 
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"
Authenticate using the service account: 
gcloud auth activate-service-account --key-file="/path/to/your/service-account-key.json"

​
Access Bucket Contents

Once authenticated, you can list and access bucket contents using gsutil: 
gsutil ls gs://your-bucket-name/
Example operations: 
# List all files in a bucket
gsutil ls gs://my-example-bucket/

# Copy files from bucket to local machine
gsutil cp gs://my-example-bucket/data.zip ./

# Copy files from local machine to bucket
gsutil cp ./local-file.txt gs://my-example-bucket/

# Sync directories
gsutil rsync -r ./local-folder gs://my-example-bucket/remote-folder

​
Integration with Visual Layer

After setting up access to your GCS bucket, you can run Visual Layer’s profiler using either method.

​
Using Mounted Bucket

With the bucket mounted as a local filesystem, run the profiler directly on the mounted directory: 
# Ensure bucket is mounted
gcsfuse my-example-bucket ~/gcs_mount

# Run Visual Layer profiler on the mounted folder
./run_profiler.sh --input ~/gcs_mount/dataset-folder/

​
Using Command-Line Copied Data

First copy the data locally using gsutil, then run the profiler: 
# Copy data from bucket to local directory
gsutil -m cp -r gs://my-example-bucket/dataset-folder/ ~/local-dataset/

# Run Visual Layer profiler on the local folder
./run_profiler.sh --input ~/local-dataset/

​
Configuration Options

When using GCS buckets with Visual Layer on-premise installations, you can:
  • Use mounted buckets for seamless filesystem access (recommended).
  • Copy data locally using gsutil commands for faster processing.
  • Configure bucket URLs directly in Visual Layer dataset creation.
For Visual Layer specific configuration, refer to the S3 Configuration documentation for similar patterns that can be adapted for GCS.

Amazon S3 Integration

Configure S3 bucket access for Visual Layer

Self-Hosting Documentation

On-premise deployment guides and scripts

​
Need Help?

For additional support: