Skip to content

[VDP-36907] Label Microsoft Fabric requests #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
GregHetherington merged 2 commits into from
Sep 12, 2025
Merged

[VDP-36907] Label Microsoft Fabric requests #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9d8055d
Label Fabric requests
GregHetherington Sep 5, 2025
937d8ba
added data source param
GregHetherington Sep 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# vena-etl-python-interface
Python package for interacting with Vena's ETL APIs. This is the Official Vena package
Python package for interacting with Vena's ETL APIs.

## Installation

Expand All @@ -16,8 +16,8 @@ HUB = 'eu1' # e.g., us1, us2, ca3
API_USER = 'your_api_user'
API_KEY = 'your_api_key'
TEMPLATE_ID = 'your_template_id'
DATA_SOURCE = 'your_erp_or_data_source'
MODEL_ID = 'your_model_id' # Optional, needed for exports
JOB_TEMPLATE_ID = 'your_job_template_id' # Template ID for job operations
```

## Usage
Expand All @@ -33,7 +33,8 @@ vena_etl = VenaETL(
api_user=API_USER,
api_key=API_KEY,
template_id=TEMPLATE_ID,
model_id=MODEL_ID
data_source=DATA_SOURCE
model_id=MODEL_ID,
)
```

Expand Down
2 changes: 1 addition & 1 deletion vepi/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Vepi - A Python package for interacting with Vena's ETL and data export APIs.
"""

from .version import __version__
from .vena_etl import VenaETL

__version__ = "0.1.3"
__all__ = ["VenaETL"]
24 changes: 21 additions & 3 deletions vepi/vena_etl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import os
from datetime import datetime
from io import StringIO
from .version import __version__
import json

class VenaETL:
Expand All @@ -54,10 +55,11 @@ class VenaETL:
api_user (str): API user from Vena authentication token
api_key (str): API key from Vena authentication token
template_id (str): ETL template ID
data_source (str): Please indicate the ERP or data source, this helps Vena with support and troubleshooting.(e.g., NetSuite, Dynamics 365, SAP, ADP, Other)
Copy link

@MxSahil MxSahil Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you feel about making a bunch of string constants or making this an enum? Standardizing these values would make triaging and gathering metrics easier. For example, one customer might use intacct and another might use sage but they're both the same source

Copy link
Contributor Author

@GregHetherington GregHetherington Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only issue with this is if we don't have an Enum for the all the different systems(which there are too many to list). This will be used to help us understand issues with the library or import jobs in Vena.

Copy link

@MxSahil MxSahil Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I was thinking we do something like

enum DataSources { 
NETSUITE,
INTACCT,
// If your source is not listed here then add it
}

This way we standardize the most common sources and then in future updates we can extend this to include the common enums that users create themselves

Copy link
Contributor Author

@GregHetherington GregHetherington Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh did you mean we should recommend sources names(with the ENUM), but don't add them to Repo as mandatory, so its still a string?

Copy link

@MxSahil MxSahil Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that works too. Just want to give the customer a list of standardized names, whether that's through an enum, list of string constants or some other way

Copy link
Contributor Author

@GregHetherington GregHetherington Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think I'll update some of the sample code in the ReadMe with the most common ones in a follow up PR. I don't want it to be strict, and tbh some users won't want to share their data source, which is fine.

model_id (str, optional): Model ID for export operations
"""

def __init__(self, hub: str, api_user: str, api_key: str, template_id: str, model_id: Optional[str] = None):
def __init__(self, hub: str, api_user: str, api_key: str, template_id: str, data_source: str, model_id: Optional[str] = None):
"""
Initialize the Vena ETL client.

Expand All @@ -66,10 +68,15 @@ def __init__(self, hub: str, api_user: str, api_key: str, template_id: str, mode
api_user (str): API user from Vena authentication token
api_key (str): API key from Vena authentication token
template_id (str): ETL template ID
data_source (str): Please indicate the ERP or data source, this helps Vena with support and troubleshooting.(e.g., NetSuite, Dynamics 365, SAP, ADP, Other)
model_id (str, optional): Model ID for export operations

"""
if not all([hub, api_user, api_key, template_id]):
raise ValueError("hub, api_user, api_key, and template_id are required")

if not data_source:
raise ValueError("data_source is required. Please indicate the ERP or data source, this helps Vena with support and troubleshooting")

self.hub = hub
self.api_user = api_user
Expand All @@ -84,15 +91,26 @@ def __init__(self, hub: str, api_user: str, api_key: str, template_id: str, mode
self.create_job_url = f'{self.base_url}/etl/templates/{template_id}/jobs'
self.job_status_url = f'{self.base_url}/etl/jobs' # Base URL for job operations
self.intersections_url = f'{self.base_url}/models/{model_id}/intersections' if model_id else None


def get_user_agent():
lib_name = "vena-etl-python-interface"
lib_version = __version__
try:
from notebookutils import mssparkutils
return f"{lib_name}/{lib_version}; platform/Microsoft Fabric; source/{data_source}"
except ImportError:
return f"{lib_name}/{lib_version}; platform/Other; source/{data_source}"

# Headers for requests
self.headers = {
"accept": "application/json",
"content-type": "application/json",
"User-Agent": get_user_agent(),
}

self.file_headers = {
"accept": "application/json",
"User-Agent": get_user_agent(),
}

def _validate_dataframe(self, df: pd.DataFrame, required_columns: Optional[List[str]] = None) -> None:
Expand Down Expand Up @@ -371,7 +389,7 @@ def export_data(self, page_size: int = 100000) -> Optional[pd.DataFrame]:
Export intersections data from the Vena model with pagination support.

Args:
page_size (int): Number of records to fetch per page (default: 100000)
page_size (int): Number of records to fetch per page (default: 5000, max: 100000)

Returns:
Optional[pd.DataFrame]: DataFrame containing all intersections data, or None if there was an error
Expand Down
1 change: 1 addition & 0 deletions vepi/version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__version__ = "0.1.3"