Thu, 23 Sep 2021 23:44:52 GMT

.
├── Cargo.lock
├── Cargo.toml
└── src
   ├── cli.yaml
   ├── commands
   │   ├── js_to_ts.rs
   │   └── mod.rs
   └── main.rs

Under src we have the cli.yaml file. This holds the definition of our application and it's commands. At link below, you'll see a command to change Javascript files to a Typescript file simply by changing the extension. There is an option to specify a directory to search recursively.

cli-example/cli.yaml at main · mitchgollub/cli-example

Contribute to mitchgollub/cli-example development by creating an account on GitHub.

GitHubmitchgollub

main.rs holds the application orchestration code. You'll see some code that translates string input to a Command. The program first runs some setup code to configure logging and error reporting. After that, it parses the CLI input and runs the appropriate command.

cli-example/main.rs at main · mitchgollub/cli-example

Contribute to mitchgollub/cli-example development by creating an account on GitHub.

GitHubmitchgollub

The final file of note is the js_to_ts.rs file under the commands folder. This is all the code that runs when the command is selected. The file here shows a simple command that searches a folder recursively and changes any files with a .js extension to a .ts extension. It uses tokio tasks to leverage some system concurrency.

cli-example/js_to_ts.rs at main · mitchgollub/cli-example

Contribute to mitchgollub/cli-example development by creating an account on GitHub.

GitHubmitchgollub

Checkout the full source and give it a ⭐ on Github if you've found it useful. Thanks for reading! 📖

]]>

Fri, 18 Jun 2021 12:47:00 GMT

.
├── Cargo.lock
├── Cargo.toml
├── events // folder to hold Lambda JSON events
│   ├── google.json
│   └── turnip-exchange.json
├── README.md
├── run.sh // script to run Lambda code locally 
├── src
│   ├── client
│   │   └── mod.rs // surf client code
│   └── main.rs // AWS Lambda Rust runtime code
└── template.yaml // SAM template for Lambda

main.rs has the code the sets up the AWS Lambda Rust runtime. There are more examples and explanations on this code at the links in the previous section.

mod client; // module for surf client wrapper

use client::RustLambdaHttpClient;
use lambda::{handler_fn, Context}; // AWS Lambda runtime for Rust
use serde_derive::{Deserialize, Serialize};

type Error = Box;

/// The custom incoming Lambda event structure
#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase")]
struct CustomEvent {
    host: String,
    path: String,
    http_verb: String,
    #[serde(default)]
    post_data: String,
}

/// The custom outgoing Lambda event structure
#[derive(Serialize)]
struct CustomOutput {
    message: String,
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    // shared client to be used across lambda invocations
    let client = RustLambdaHttpClient::new();
    let client_ref = &client;
    
    // entry point for the lambda event
    lambda::run(handler_fn(
        move |event: CustomEvent, _ctx: Context| async move {
            dbg!(&event);
            let url = format!("{host}{path}", host = event.host, path = event.path);
            let body = client_ref
                .clone()
                .send(event.http_verb, url, event.post_data)
                .await?;

            Ok::(CustomOutput {
                message: format!("Body: {}", body),
            })
        },
    ))
    .await?;

    Ok(())
}

Making an HTTPS request

A working sample of Rust running on Lambda is available on my Github. It parses an incoming event to make a simple HTTP request using surf. You can run the lambda locally in a docker container using the run.sh script available in the repository. Deployment to your AWS environment can be done through the SAM CLI tool from AWS.

I've separated the client code to a client module. The code for that is below

use crate::Error;
use surf::http::Method;
use surf::Client;
use surf::Url;

/// Wrapper used to share the surf client struct
#[derive(Clone)]
pub struct RustLambdaHttpClient {
    client: Client,
}

impl RustLambdaHttpClient {
    pub fn new() -> Self {
        RustLambdaHttpClient {
            client: surf::Client::new(),
        }
    }

    pub async fn send(
        self,
        http_verb: String,
        url: String,
        post_data: String,
    ) -> Result {
        let mut res = self.client
            .send(
                surf::RequestBuilder::new(parse_http_method(&http_verb), Url::parse(&url)?)
                    .body(post_data)
                    .header(
                        "User-Agent",
                        "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0",
                    )
                    .build(),
            )
            .await?;
        let body = res.body_string().await?;
        dbg!(&body);
        Ok(body)
    }
}

/// Parses HTTP method codes from the Lambda Event
fn parse_http_method(input_method: &str) -> Method {
    match input_method {
        "GET" => Method::Get,
        "POST" => Method::Post,
        _ => panic!("No matching HTTP method for {}", input_method),
    }
}

Source Code

My source code is below! Feel free to use it as a template for your own needs. Hopefully it helps. Since I'm new to Rust, any feedback is appreciated.

mitchgollub/rust-lambda-https

Contribute to mitchgollub/rust-lambda-https development by creating an account on GitHub.

GitHubmitchgollub

Package.json

I had to edit the package.json file in the SAM template to make this work. Below are the npm start and npm run debug commands I created to run the SAM environment with NPM.

	{
	"name": "hello_world",
	"version": "1.0.0",
	"description": "hello world sample for NodeJS",
	"main": "app.js",
	"repository": "https://github.com/awslabs/aws-sam-cli/tree/develop/samcli/local/init/templates/cookiecutter-aws-sam-hello-nodejs",
	"author": "SAM CLI",
	"license": "MIT",
	"scripts": {
	"debug": "sam local invoke --template ../template.yaml --event ../events/event.json -d 9229",
	"start": "sam local invoke --template ../template.yaml --event ../events/event.json"
	}
	}

view raw package.json hosted with ❤ by GitHub

The most important sections are lines 9 and 10. These will actually launch the SAM CLI tooling instead of Node.js. The debug script will specifically be used to hook into the VS Code Debugger tools.

Launch.json

This is a crucial configuration for setting up these two tools to work together. The launch.json is a file that VS Code uses when setting up the debugger. I have my code posted below, and I'll walk through the important parts.

	{
	"version": "0.2.0",
	"configurations": [
	{
	"type": "node",
	"request": "launch",
	"name": "Launch via NPM",
	"runtimeExecutable": "npm",
	"cwd": "${workspaceFolder}/hello-world",
	"runtimeArgs": [
	"run-script",
	"debug"
	],
	"port": 9229,
	"skipFiles": [
	"/**"
	],
	"localRoot": "${workspaceRoot}/hello-world",
	"remoteRoot": "/var/task"
	}
	]
	}

view raw launch.json hosted with ❤ by GitHub

The type is node, so if you are using a different programming language, you'll need to use an appropriate configuration in VS Code. On line 9 is cwd or current working directory. You'll need this to work with the application structure generated by the SAM CLI.

For debugging, the lines to add are 18 and 19 for localRoot and remoteRoot. This will allow VS Code to map your local JS files to the JS files running in the Lambda container.

Once you have the files set up, give it a try! Set a breakpoint in app.js and launch the VS Code Debugger. You should see the execution pause on your breakpoint and have the ability to inspect variables.

Hope this was helpful! If it was or you want to make some suggestions, feel free to send me an email at mitch.gollub@gmail.com.

]]>

Fri, 15 Nov 2019 16:23:53 GMT

Over the past few weeks I've been having trouble finding time to write code. It's usually a consequence of not being right at my desk at all times with my desktop machine. It doesn't help that my laptop is a 15" Macbook from 2013, so it's not the most comfortable hunk of metal to lug around the house.

However, my 10" Chromebook is! I decided to see if it was possible to use this little guy as a lightweight coding machine. After doing some online research, I found these tools to be immensely helpful in getting a Development Environment set up for node.

Notes on Crouton

One popular option I found for creating a development environment on a Chromebook is to use a Linux distribution. While I'd most likely prefer using Ubuntu or another type of Linux OS for development, my goal was to get an environment on the smallest system I own. That being said, I had two problems with the Crouton approach: security and performance. Crouton requires running ChromeOS in Developer Mode. While it's relatively harmless to do this, I do enjoy the sandbox security that comes baked into ChromeOS. In addition to security concerns, I'd most likely experience pretty poor performance on my Chromebook's rudimentary hardware. With that, let's move on to the tools I chose to use.

Termux

Termux is a Linux terminal emulator for Android devices. If your chromebook can run Android apps, then this should work for you. It runs pretty well on my ASUS C100 with a Rockchip processor and 2GB of RAM.

Text Editors

It's worth installing a text editor as well. Termux comes with Vim and has a package for Nano, so those can be used if you're interested. For those not so comfortable on the command line, Caret is a good option. However there are countless other text editors for Chromebooks you can try.

Okay, now we're ready for some development! Let's get Termux configured and try to launch a simple react app!

Install Node

Termux has it's own package manager. You can search that list by typing pkg search or view the whole list with pkg list-all. You can also get this info by visiting their page.

At the time of writing, Termux offers two NodeJS installations: nodejs and nodejs-lts. I'm going to play it safe and install the LTS pkg, but either will work just fine. Go ahead and install with pkg install nodejs-lts

Creating a React App

Installing the NodeJS runtime will also give you the NPM package manager! Unfortunately there are some issues around running create-react-app in Termux. I was able to reference this stackoverflow article to get it to work. Follow the steps below to create a project called chromebook-react.

npm cache clean --force
mkdir chromebook-react
touch chromebook-react/.gitignore
npx create-react-app chromebook-react

On a slower machine like mine, this may take a while. In the meantime, feel free to pick up an energizing hot beverage ☕

Running the Development Server

Great! Now you can set up the development server for React the same way you would normally: npm start.

There is a quirk about running the server in the Termux emulator. Even though the displayed address will be something like http://localhost:3000, your Chrome browser will not be able to access the server via localhost. You will need to get the IP of the Termux terminal.

You can do this by typing ipconfig in the window and seeing Termux's IP address. It's been a consistent value of 100.115.92.2 at the time of writing this. So, once the server is up, you should be able to access the React page by typing http://100.115.92.2:3000/ in your browser!

And that's it! You should be able to edit some files in the React project and see them change on save in your browser. This environment won't work for heavier workloads, but it's really nice to have the flexibility to do development work on a cheap 11" chromebook.

I hope you found this useful! Please feel free to share your thoughts with me at mitch.gollub@gmail.com

]]>

Sat, 28 Sep 2019 02:39:51 GMT

const HoC = Component => EnhancedComponent;

view raw react-high-order-components.js hosted with ❤ by GitHub

An example for using a HOC would be if you want the same attribute to appear on multiple components such as className. You could write a HOC such as the one below:

	const withClassName = Component => props => (

	);

view raw react-high-order-components-2.js hosted with ❤ by GitHub

Note the "with" prefix on the name of the HOC. It is common practice to prefix HOCs that provide some information to the components they enhance using this pattern.

Recompose Library

The Recompose library provides many useful HOCs as small utilities that we can use to wrap our components, moving away from some of their logic.

Check out some of these useful HOCs:

flattenProp()

Sometimes the objects we work with are structurally complex. You might want to flatten the structure of incoming objects to make them easier to work with in your component.

	const Profile = ({ username, age }) => (

	Username: {username}
	Age; {age}

	);
	Profile.propTypes = {
	username: string,
	age: number
	};

	// Usage - wrapping the component with the function
	const withFlattenUser = flattenProp('user');
	const ProfileWithFlattenUser = withFlattenUser(Profile);

	// OR
	const ProfileWithFlattenUser = flattenProp('user')(Profile);

view raw react-high-order-components-3.jsx hosted with ❤ by GitHub

renameProp()

The renameProp() HOC allows for you to rename a passed prop for use in the component.

	const Profile = ({ name }) => (

	Name: {name}

	);
	Profile.propTypes = {
	name: string
	};

	const ProfileWithRenamedUsername = renameProp('username', 'name')(Profile);

view raw react-high-order-components-4.jsx hosted with ❤ by GitHub

You can also combine HOCs to constructed an enhanced component for you application

	const enhance = compose(
	flattenProp('user'),
	renameProp('username', 'name')
	);

	// Combine HOC's (from Recompose or custom) with compose
	const EnhancedProfile = enhance(Profile);

view raw react-high-order-components-5.jsx hosted with ❤ by GitHub

Performance

Remember to use HOCs with caution. While they make code much easier to read, the trade off for using HOCs is in performance. Wrapping a component into a higher order one adds a new render function, a new life cycle method call, and memory allocation.

]]>

Fri, 16 Aug 2019 01:58:41 GMT

	Object shares = getValueFromField("SHARES");
	Shares = (string)shares;

view raw defensive-programming-strings-1.cs hosted with ❤ by GitHub

The Object is the ultimate base class used by classes in .NET. This means we could be returning ANY class from the getValueFromField() function and try to cast it into a string. If a type other than System.String is retrieved, you might get an error such as this one: System.InvalidCastException: Unable to cast object of type 'System.Double' to type 'System.String'.

.NET has a .ToString() method that is much safer to use than a type cast. The method is implemented on many classes, ensuring that you're much more likely to guarantee that you can produce a String type. Below was what we introduced to allow for processing to continue. Note the null-conditional operator. You can read more on that here: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/member-access-operators#null-conditional-operators--and-

	Object shares = getValueFromField("SHARES");
	Shares = shares?.ToString();

view raw defensive-programming-strings-2.cs hosted with ❤ by GitHub

It looks like the data type for that incoming field had changed to being interpreted as a Double at some point. However, we still want to write that field to the output text file. Making this change allowed the file to be processed without error. While the program runs, ideally we should leverage the strong typing in C# and use a new method like getDoubleFromField() to make sure we aren't passing our logic a generic Object class, which was the root cause of the issue. The reason we did not, is because there is no processing that really requires the System.Double type for this field. Simply going from a string field on the Excel sheet to a String in the output file was the most defensive option.

There's so much more to talk about on this topic, so I'll be updating it as I write more. Stay tuned for my upcoming sections on parsing String data. If you liked what you read, let me know! Feel free to reach out to me at mitch.gollub@gmail.com. Happy coding!

]]>

Sat, 25 May 2019 15:40:48 GMT

	Write-Host "Environment Variable Substitution"

	$variables = gci $env:$env:APPLICATION_PREFIX* \| Select-Object -Property Key, Value

	$configPath = "$env:APPLICATION_PATH\$env:CONFIG_FILE"

	Write-Output "Loading config file from $configPath"
	$appSettings = Get-Content -Raw $configPath \| ConvertFrom-Json

	foreach($variable in $variables) {
	$matchString = $variable.Key.replace($env:APPLICATION_PREFIX, "")

	$matchProperties = $matchString.Split(".")

	if($matchProperties.Count -gt 1) {
	$match = $appSettings.($matchProperties[0]).psobject.properties \| where { $_.Name -eq $matchProperties[1] }
	if ($match) {
	$appSettings.($matchProperties[0]).($matchProperties[1]) = $variable.Value
	}
	else {
	Write-Output "Could not find match for $matchString"
	}
	}
	else {
	$match = $appSettings.psobject.properties \| where { $_.Name -eq $matchString }
	if ($match) {
	$appSettings.($matchString) = $variable.Value
	}
	else {
	Write-Output "Could not find match for $matchString"
	}
	}
	}

	$appSettings \| ConvertTo-Json -depth 100 \| Out-File $configPath

view raw net-core-appsettings-json-variable-replacement-in-appveyor-1.ps1 hosted with ❤ by GitHub

The script utilizes the user-defined environment variables that AppVeyor writes to the target machine for deployments. I use a prefix ($env:APPLICATION_PREFIX) for grabbing an Array of those variables, then loop through them. Each iteration of the loop searches the target appsettings.json file (or any other JSON based configuration file you feed it) for matching properties. If a property matches the name of an environment variable (with the environment prefix stripped out), then the value of the environment variable will be written to the output file.

Now what this doesn't support that you can find in Octopus Deploy is the array variable substitution. Currently this handles root level and hierarchical property values, but I'd like to add array support some time in the future. I'd also like to expand this to use recursion so it can support more than a property depth of one.

Feel free to use this script for your own purposes. I had trouble finding one like it, so hopefully you find this suitable for what you need to get started. If you find a way to improve it, I'd love to hear about it! Send an email to mitch.gollub@gmail.com. Thanks for reading!

UPDATE 1/31/21:

If you've been having trouble with the above script, @eivindivine was able to make some adjustments and get it working. He shared his example here:

	Write-Host "Environment Variable Substitution"

	$prefix = ${env:APPLICATION_PREFIX} + "*"

	$props = gci env:$prefix \| Select-Object -Property Name, Value

	$configPath = "$env:APPLICATION_PATH\$env:CONFIG_FILE"

	Write-Output "Loading config file from $configPath"
	$appSettings = Get-Content -Raw $configPath \| ConvertFrom-Json

	foreach($variable in $props) {

	$matchString = $variable.Name.replace(${env:APPLICATION_PREFIX}, "")

	$matchProperties = $matchString.Split(".")

	if($matchProperties.Count -gt 1) {
	$match = $appSettings.($matchProperties[0]).psobject.properties \| where { $_.Name -eq $matchProperties[1] }
	if ($match) {
	$appSettings.($matchProperties[0]).($matchProperties[1]) = $variable.Value
	}
	else {
	Write-Output "Could not find match for $matchString"
	}
	}
	else {
	$match = $appSettings.psobject.properties \| where { $_.Name -eq $matchString }
	if ($match) {
	$appSettings.($matchString) = $variable.Value
	}
	else {
	Write-Output "Could not find match for $matchString"
	}
	}
	}

	$appSettings \| ConvertTo-Json -depth 100 \| Out-File $configPath

view raw gistfile1.txt hosted with ❤ by GitHub

Check out the link to his gist for more information on what changed!

]]>

Sat, 01 Dec 2018 14:42:21 GMT

Summary: Write code to consume data in a CSV file, and generate reports.

You may use any language, libraries or tools you want, as long as you will be
able to demonstrate your solution in person. Code that you can email to the
interviewers ahead of time usually works best, but other means of
demonstration, such as a laptop with the necessary tools loaded, would be fine
as well.

Input Specification: CSV file. First line is a header. Columns are:
    TXN_DATE - date transaction took place
    TXN_TYPE - type of transaction (BUY/SELL)
    TXN_SHARES - number of shares affected by transaction
    TXN_PRICE - price per share
    FUND - name of fund in which shares are transacted
    INVESTOR - name of the owner of the shares being transacted
    SALES_REP - name of the sales rep advising the investor

Output Specification:

    1. Provide a Sales Summary:
        For each Sales Rep, generate Year to Date, Month to Date, Quarter to
        Date, and Inception to Date summary of cash amounts sold across all
        funds.

    2. Provide an Assets Under Management Summary:
        For each Sales Rep, generate a summary of the net amount held by
        investors across all funds.

    3. Break Report:
        Assuming the information in the data provided is complete and accurate,
        generate a report that shows any errors (negative cash balances,
        negative share balance) by investor.

    4. Investor Profit:
        For each Investor and Fund, return net profit or loss on investment.

Be prepared to discuss:

    Testing strategies
    Difficulties you had and how you solved them

Suggestion:

    Your code should be correct first, have good style and comments second, and
    then be clever/fast as a distant third.

The Code

I chose to write the project with a .NET Core back-end to consume the CSV Data and an Angular front-end to display the reports. The .NET Core server code uses the CSV Helper library to consume and generate strong-typed entities from the CSV file. I do this here in the code: https://github.com/mitchgollub/DotNetCore.Angular.SalesCSVReader/blob/05c78f9f3dd7b130e9ad566dc387b4ff7bf762ec/Repositories/TransactionRepository.cs#L24-L34

CSV Helper is written and maintained by Josh Close and is taking PR's on GitHub. More info on CSV Helper here: https://joshclose.github.io/CsvHelper/.

Writing the reports in Angular helped me easily grab aggregated data from the back-end server and display it in tables on different pages. I could define a ViewModel for each report with all of the completed columns and have the server send over the rows. That happens here: https://github.com/mitchgollub/DotNetCore.Angular.SalesCSVReader/blob/05c78f9f3dd7b130e9ad566dc387b4ff7bf762ec/ClientApp/src/app/sales-rep-summary/sales-rep-summary.component.ts#L11-L23

Things I would have done differently

So, given that the CSV was rather short, I had not run into any performance issues running the solution as it was. However, the architecture would easily crawl to a halt when aggregating a large dataset. There are two changes I could make to allow the application to scale: Reduce iterations over the dataset and minimize the amount of times the code reads the CSV file.

One example of the iterations I did here shows four aggregates that pass over the data multiple times.

	foreach (var salesRep in salesReps)
	{
	var summary = new SalesSummary { SalesRep = salesRep };


	var repTransactions = transactions.Where(x => x.SalesRep == salesRep);


	summary.Y2DSold = repTransactions.Where(x => x.Date >= new DateTime(dateTimeNow.Year, 1, 1))
	.Select(x => CalculateTransaction(x))
	.Sum();


	summary.M2DSold = repTransactions.Where(x => x.Date >= new DateTime(dateTimeNow.Year, dateTimeNow.Month, 1))
	.Select(x => CalculateTransaction(x))
	.Sum();


	summary.Q2DSold = repTransactions.Where(x => GetQuarter(x.Date) == GetQuarter(dateTimeNow))
	.Select(x => CalculateTransaction(x))
	.Sum();


	summary.I2DSold = repTransactions
	.Select(x => CalculateTransaction(x))
	.Sum();


	salesSummaries.Add(summary);
	}

view raw c-sharp-and-csvs-1.cs hosted with ❤ by GitHub

I can reduce the number of passes through the data by leveraging a LINQ Aggregate. The Aggregate calculation will be heavy in that it would calculate the total monetary value from the transaction using the CalculateTransaction function, then check each transaction for a date matching date range to be added to the totals for Year-to-Date, Month-to-Date, etc. However, with a significantly large dataset, the performance benefit of passing the data only once will outweigh the extra computation. More on the Aggregate function here: https://docs.microsoft.com/en-us/dotnet/api/system.linq.enumerable.aggregate?view=netcore-2.1

Another point I would improve would be the code reuse on the Angular app. Each component is a copy and paste job from the original .NET Core/Angular template project from Microsoft. Given more time, I'd break out the HttpClient calls to a ReportDataService that could be injected to each component. Then there could be a function on the ReportDataService that calls a wrapper function around the reused HttpClient.get() commands. I could even merge the .NET Core endpoints into one Controller, return all the report aggregate data in one ViewModel, and just make one HttpClient call. Merging the endpoints would reduce code duplication in the .NET Core app as well.

The final major update that could be implemented would be some graceful failing in the event that the CSV structure changes. As the app stands at the time of writing this, it is semi-resilient to CSV changes. If new columns are added, they will simply not be picked up by the code. However, if the CSV file moves, the existing column names change, or existing columns are removed, the server will throw a 500 error when trying to return a report.

Implementing this resilience might be a little more involved. I'd imagine it'd be good to store a backup CSV file from the last successful run of the application. That way if the file has an error during data retrieval, we can fall back on the backup file. This will hide the issue, so proper logging and notifications should be put in place to alert the party involved with the maintenance of the CSV file can make adjustments. If the business need requires showing that the file is ill-formatted, we should bubble that message up to the UI (in a friendly way) so the user is aware and can handle the situation accordingly.

In short, working with CSV files in .NET Core isn't so bad. CSVHelper made reading the file incredibly easy. I'd still prefer a database to store the data, but for some custom integrations, CSV's are a very real part of people's workflows. The project itself was impossible to complete in a short amount of time, but I'm happy with what I was able to make. One day maybe I'll go back and it round out with the items I listed that I would change.

]]>

Sat, 03 Nov 2018 18:18:27 GMT

Making an HTTPS request

Source Code

Other Links

Package.json

Launch.json

Notes on Crouton

Termux

Text Editors

Install Node

Creating a React App

Running the Development Server

Recompose Library

Performance

UPDATE 1/31/21:

The Code

Things I would have done differently