I’ve been using Bitnami’s redis server on the default NodeJS instance on Lightsail but unfortunately, it’s not the full stack so it’s missing GEO, JSON and full text search support which is a bit of a bummer. So, to solve that problem I decided to see if I could setup Redis Stack using docker on a Lightsail instance. I decided not to use LS container support as I can get an 8GB instance for $40/month, vs. a similar container instance at $100/month.
Redis Stack Cloud on us-west-2 for 5GB of RAM runs $105/month (as of April ’23) without HA.
To start, I created a Debian OS only instance and installed docker:
Update May 8, 2023 I found that the various modules included as part of Redis Stack were not loading due to my overriding redis.conf so I had to add the following to my conf file:
I use the 511.org API and one question I see come up frequently on the support forum are requests to increase the number of API requests possible using an API key. The answer is pretty straightforward, the 511 API is not intended to serve as your application’s backend. If you want to serve the data to large numbers of users you’ll need to use your API key, fetch the data, cache it on your own backend and serve it from there. You can then use your API key to refresh your own cached data and serve your application.
Another question that often comes up is access to historic data which, as of this writing, the API simply doesn’t provide. I would doubt the 511.org team will offer historic data anytime soon (if ever). If you’re looking for historic data the answer would be to build your own backend and archive the data.
To request an increased rate limit, you can provide the following information to transitdata@511.org:
your API key
the rate limit you need
a description of the use case
data endpoints you intend to use
a brief justification for the requested rate limit
I’ve worked with a lot of DOT type data and I have to say Massachusetts takes the cake for a seemingly unnecessarily complicated authentication process. The processes is documented on their /auth-token API.
This endpoint is available to all authorized users. The content of the response is the same for all user groups. The Bearer signature generation that uses this token is performed as follows.
MADOT
Ok, here is the six, yes six step process to authenticate MASSDOT APIs:
Step 1: Concatenate the user’s secret key with the token provided herein, separated by a colon. (‘SecretKey:token’)
Step 2: Create a SHA256 hash of the concatenated string. (i.e. SHA256(secretKey:token))
Step 3: Convert the generated hash to a hexadecimal string representation
Step 4: Concatenate the user’s username with the hexadecimal hash, separated by a colon. (‘username:hexadecimalHash’)
Step 5: Create the base-64 string representation of the concatenation. (Base64(‘username:hexadecimalHash’))
Step 6: The result is the signature required for Bearer type authorization. (Authorization Bearer ‘generated signature’)
Btw, I provided this gist to MASSDOT in the event they want to link/share for other developers.
I’ve been exploring the Apollo stack for developing with GraphQL and find the documentation a bit outdated so I decided to make some notes for myself and start collecting them here. The first thing I wanted to do is experiment with the apollo client codegen for TypeScript and understand how this tool works and leverage it for creating a TypeScript Apollo client. I started by using this Starwars sample Apollo server so I could focus on the client-side code gen which was quick and easy to stand up.
$ git clone https://github.com/apollographql/starwars-server.git
...
$ cd starwars-server
$ yarn && yarn start
yarn run v1.15.2
$ nodemon ./server.js --exec babel-node
[nodemon] 1.19.0
[nodemon] to restart at any time, enter `rs`
[nodemon] watching: *.*
[nodemon] starting `babel-node ./server.js`
🚀 Server ready at http://localhost:8080/graphql
🚀 Subscriptions ready at ws://localhost:8080/websocket
Next, I tested a simple GraphQL query to make sure the server is working by browsing here:
http://localhost:8080/graphql
I installed the Apollo CLI and started experimenting with codegen. Unfortunately, as of this writing the CLI documentation is outdated and refers to apollo-codegen and the parameters and configuration appear to have changed. To play with newer apollo CLI and client-side codegen I created a new “project” folder and just wanted to get some code generated without any other project dependencies/files etc. So, I created a folder to get started:
$ mkdir starwars-client
$ cd starwars-client
Next, I ran the apollo CLI to download the server’s schema with –endpoint parameter pointing to the running instance of the starwars-server sample:
$ ➜ starwars-client apollo client:download-schema --endpoint=http://localhost:8080/graphql
⚠️ It looks like there are 0 files associated with this Apollo Project. This may be because you don't have any files yet, or your includes/excludes fields are configured incorrectly, and Apollo can't find your files. For help configuring Apollo projects, see this guide: https://bit.ly/2ByILPj
✔ Loading Apollo Project
✔ Saving schema to schema.json
$ ls
schema.json
$
As you can see, this created a schema.json file containing details from my starwars-server. The next step is generating TypeScript code for a single GraphQL query using the downloaded schema. For good measure I’ll include a few of the issues I ran into along the way as I didn’t fine a lot on Google related to the various error messages.
➜ starwars-client apollo client:codegen
› Error: Missing required flag:
› --target TARGET Type of code generator to use (swift | typescript | flow | scala)
› See more help with --help
Ok, so I’m missing –target, that’s easy enough to add…
➜ starwars-client apollo client:codegen --target typescript
Error: No schema provider was created, because the project type was unable to be resolved from your config. Please add either a client or service config. For more information, please refer to https://bit.ly/2ByILPj
at Object.schemaProviderFromConfig (~/.nvm/versions/node/v10.15.3/lib/node_modules/apollo/node_modules/apollo-language-server/lib/providers/schema/index.js:29:11)
at new GraphQLProject (~/.nvm/versions/node/v10.15.3/lib/node_modules/apollo/node_modules/apollo-language-server/lib/project/base.js:31:40)
at new GraphQLClientProject (~/.nvm/versions/node/v10.15.3/lib/node_modules/apollo/node_modules/apollo-language-server/lib/project/client.js:33:9)
at Generate.createService (~/.nvm/versions/node/v10.15.3/lib/node_modules/apollo/lib/Command.js:114:28)
at Generate.init (~/.nvm/versions/node/v10.15.3/lib/node_modules/apollo/lib/Command.js:37:14)
➜ starwars-client
Again, unfortunately the bitly short link provided by the tool points back to the outdated apollo-codegen documentation which is inaccurate. So I added –localSchemaFile pointing to my newly downloaded schema.json:
➜ starwars-client apollo client:codegen --localSchemaFile=schema.json --target=typescript
⚠️ It looks like there are 0 files associated with this Apollo Project. This may be because you don't have any files yet, or your includes/excludes fields are configured incorrectly, and Apollo can't find your files. For help configuring Apollo projects, see this guide: https://bit.ly/2ByILPj
✔ Loading Apollo Project
✖ Generating query files with 'typescript' target
→ No operations or fragments found to generate code for.
Error: No operations or fragments found to generate code for.
at write (~/.nvm/versions/node/v10.15.3/lib/node_modules/apollo/lib/commands/client/codegen.js:61:39)
at Task.task (~/.nvm/versions/node/v10.15.3/lib/node_modules/apollo/lib/commands/client/codegen.js:86:46)
➜ starwars-client
What this error is actually saying is that the tool is expecting to find either .graphql or .ts files that have GraphQL “operations” aka queries, or mutations defined within my project folder which I haven’t created yet. Turns out there are a few options, 1) create .ts files with gql constants or 2) create a .graphql file(s) that contain named queries. I started with a simple query.graphql file for testing like this:
query {
heros(episode: NEWHOPE) {
name
}
}
I then ran the command again:
➜ starwars-client apollo client:codegen --localSchemaFile=schema.json --target=typescript
…and this yielded the same error as above because the CLI defaults to looking in ./src although you can change this using the –includes parameter. So I created the folder, moved the query.graphql file and re-ran the tool:
➜ starwars-client apollo client:codegen --localSchemaFile=schema.json --target=typescript
✔ Loading Apollo Project
✖ Generating query files with 'typescript' target
→ Apollo does not support anonymous operations
GraphQLError: Apollo does not support anonymous operations
Basically, this is telling me didn’t “name” the query so back to editing the query.graphql file and adding “heros”:
query heros {
hero(episode: NEWHOPE) {
name
}
}
Ok, now let’s try that again:
➜ starwars-client apollo client:codegen --localSchemaFile=schema.json --target=typescript
✔ Loading Apollo Project
✔ Generating query files with 'typescript' target - wrote 2 files
Success! I now have a few new folders and files added to my “project”:
In the above example I use command-line options although the apollo CLI supports a config file which looks like the following located in apollo.config.js which points to a remote schema from my starwars-server instance:
Working on building mobile apps for the last several years I thought I would publish a list of some of the things I’ve learned here in the mobile trenches. Without further adieu and in no particular order… Btw, welcome your feedback/additions. Continue reading Lessons Learned in a Mobile Startup→
Part of building mobile web apps is understanding the myriad of mobile analytics and in part visualizing the data to shed light on trends that my otherwise be difficult to see in tabular data or even a colorful cohort table. I’ve been building a dashboard using R, RStudio, Shiny, and Shiny Dashboards aggregating data from MSSQL, Postgres, Google Analytics, and Localytics.
Below is the main function to fetch the Localytics sample data and convert it into a data frame that’s suitable for plotting. Now, admittedly I’m not an R expert so there may well be better ways to slice this JSON response but this is a fairly straight forward approach. Essentially, this fetches the data, converts it from JSON to an R object, extracts the weeks, preallocates a matrix and then iterates over the data filling the matrix to build a data frame.
retentionDF <- function() {
# Example data from: http://docs.localytics.com/dev/query-api.html#query-api-example-users-by-week-and-birth_week
localyticsExampleJSON <- getURL('https://gist.githubusercontent.com/strefethen/180efcc1ecda6a02b1351418e95d0a29/raw/1ad93c22488e48b5e62b017dc5428765c5c3ba0f/localyticsexampledata.json')
cohort <- fromJSON(localyticsExampleJSON)
weeks <- unique(cohort$results$week)
numweeks <- length(weeks)
# Take the JSON response and convert it to a retention matrix (all numeric for easy conversion to a dataframe) like so:
# Weekly.Cohort Users Week.1
# 1 2014-12-29 7187 4558
# 2 2015-01-05 5066 NA
i <- 1
# Create a matrix big enough to hold all of the data
m <- matrix(nrow=numweeks, ncol=numweeks + 1)
for (week in weeks) {
# Get data for all weeks of this cohort
d <- cohort$results[cohort$results$birth_week==week,][,2]
lencohort <- length(d)
for (n in 1:lencohort) {
# Skip the first column using "+ 1" below which will be Weekly.Cohort (date)
m[i,n + 1] <- d[n]
}
i <- i + 1
}
# Convert matrix to a dataframe
df <- as.data.frame(m)
# Set values of the first column to the cohort dates
df$V1 <- weeks
# Set the column names accordingly
colnames(df) <- c("Weekly.Cohort", "Users", paste0("Week.", rep(1:(numweeks-1))))
return(df)
}
To make things easy I put together a gist and if you’re using R you can runGist it yourself. It requires several other packages so be sure to check the sources in case you’re missing any. Fair warning the Localytics API demo has very limited data so the chart, let’s just say simplistic however given many weeks worth of data it will fill out nicely (see example below).
