Handle partitioned and updating billing exports #2794
Conversation
| // Container. It uses the "Last Modified Time" of the file to determine which | ||
| // has the latest month-to-date billing data. | ||
| func (asbp *AzureStorageBillingParser) getMostRecentBlobs(start, end time.Time, client *azblob.Client, ctx context.Context) ([]string, error) { |
There was a problem hiding this comment.
pass entire blob info object up rather than just the name to preserve mod time info which is used later on
| blobModTime := *blob.Properties.LastModified | ||
| // Check if the blob was last modified before the file was modified, indicating that the | ||
| // file is the most recent version of the blob | ||
| if blobModTime.Before(fileInfo.ModTime()) { |
There was a problem hiding this comment.
What happens if this is not true? In other words, if the Blob file has a more recent timestamp than our local file's timestamp? Does this mean we need to go fetch the updated CSV file from storage?
^^ I don't think that scenario would ever happen though.
There was a problem hiding this comment.
Oops nevermind. After re-reading this, I see that we would proceed to download the updated Blob file! This comment is good to close.
| return nil, fmt.Errorf("failed to retrieve manifest %w", err) | ||
| } | ||
|
|
||
| log.Infof(string(manifestBytes)) |
There was a problem hiding this comment.
| log.Infof(string(manifestBytes)) |
Was this used for debugging?
There was a problem hiding this comment.
oops, yeah that is what that is
| @@ -47,27 +48,29 @@ func (asbp *AzureStorageBillingParser) ParseBillingData(start, end time.Time, re | |||
| } | |||
| ctx := context.Background() | |||
| // Example blobNames: [ export/myExport/20240101-20240131/myExport_758a42af-0731-4edb-b498-1e523bb40f12.csv ] | |||
There was a problem hiding this comment.
| // Example blobNames: [ export/myExport/20240101-20240131/myExport_758a42af-0731-4edb-b498-1e523bb40f12.csv ] |
Comment here is probably no longer relevant?
Signed-off-by: Sean Holcomb <[email protected]>
Sean-Holcomb
force-pushed
the
sth/partitioned-azure-billing-export
branch
from
26adf14
to
2700363
Compare
Signed-off-by: Sean Holcomb <[email protected]>
Signed-off-by: Sean Holcomb <[email protected]>
What does this PR change?
This PR updates the Azure storage Cloud Cost to allow it to work with billing exports with various setting for partitioned and file overwrite. Until now only unpartitioned non-overwriting billing exports were supported.
To support file overwrite we need to make sure that we are redownloading a blob if there have been updates to it since the last time we downloaded it. This is a simple of process of comparing the mod time of the blob and the local version if it has already been downloaded.
To support partitioned exports we look for the presence of a
manifest.jsonfile which contains a list of the blobs that contain the partitions, if this is located instead of adding the newest file for a month we add all blobs named in the manifest.Does this PR relate to any other PRs?
How will this PR impact users?
Does this PR address any GitHub or Zendesk issues?
How was this PR tested?
Does this PR require changes to documentation?
Have you labeled this PR and its corresponding Issue as "next release" if it should be part of the next OpenCost release? If not, why not?