ProjectFlow

ProjectFlow is a ad-hoc analytics project manager. Upon creating a new type Project and providing it to initiate function will crate all necessary directory into local machine to start working on the project.

Steps:

  • First we need to create a profile which ProjectFlow will use to set up initial

directories. Profile should be stored in /home/username/.projectflow/ directory. inside of .projectflow we will create a file called profiles and add the profile name and properties. Inside of priofile file should looks like this:

[default]
project_root=/home/datapsycho/JuliaProjects/adhocs
projects_dir=projects
data_dir=datalake
insights_dir=insights
insights_viz_dir=vizfiles
insights_data_dir=datafiles

The convention must be followed and all these directory map should be there. Here it say our project root where all the projects will be created. Our raw data will be stored in datalake our visuals will be stored in insights/vizfiles and our post analysis report data will be stored in insights/datafiles.

The Root Project folders must be created before going to next step. which is JuliaProjects/adhocs in the examples.

  • Next we have to create type called project and run the Initiate method.

Either it will crate a new project or it will load the paths of existing project.

using ProjectFlow

p = ProjectFlow.Project(
    id="xyz",
    name="My Fancy? *Project1 2 ",
    template="jl",
    profile="default"
)

datalake, iviz, idata = initiate(p)

The following execution will create the directories in adhocs directory such as:

JuliaProjects/adhocs/
├── datalake
│   └── 2020-07-18_My_Fancy_Project_xyz
├── insights
│   └── 2020-07-18_My_Fancy_Project_xyz
│       ├── datafiles
│       └── vizfiles
└── projects
    └── 2020-07-18_My_Fancy_Project_xyz
        └── code.jl
  • Inside of the code.jl there will already boiler plate code to start project.
# Project Name:
# Regular Imports
using ProjectFlow

p = Project(
    id="xyz",
    name="My Fancy? *Project1 2 ",
    template="jl",
    profile="default"
)

datalake, idata, iviz = initiate(p)

Which is the starting point of start writing adhoc dataanalytics single file project.

You might want to create a separate environment for whole adhocs project and an initializer jl file filled up with init code and use it every time before starting a new project.

# initializer.jl file
using ProjectFlow

p = ProjectFlow.Project(
    id="xyz",
    name="My Fancy? *Project1 2 ",
    template="jl",
    profile="default"
)

datalake, iviz, idata = initiate(p)

Internal Methods Index

ProjectFlow.ProjectType

Immutable type Project. Project hold:

  • Unique Project Id
  • Project name
  • Template extention jl tocreate .jl file, in future will have option for .ipynb notebook
  • The profile to use when creating projects, the default value is default
source
ProjectFlow.cache_templateMethod

cache_template(p::Dict, t::String)

Load the template in the memory and replace the idetifier with given value of stirng.

Argument

  • p: map of pared values of identifier and replacer
  • t: text templare to replace the identiter with value
source
ProjectFlow.clean_nameMethod

function clean_name(s::String)

Take a strign remove spacial character from the string

Argument

  • s: String to remove special character

Example:

clean_name("Here is < 12 ?")
source
ProjectFlow.consolidateMethod

consolidate(p::Project)

Consolidate a value of type project and return a Dict object with cleaned project name and newly added attribute.

#Arguemnt

  • p: Value of type project

Example

A_PROJECT = ProjectFlow.Project(
    id="xyz",
    name="My Fancy? *Project1 2 ",
    template="jl",
    profile="pycharm"
)
consolidate(A_PROJECT)
source
ProjectFlow.initiateMethod

function initiate(p::Project)

This is the main function compose all other function together and upon giving value of type project function will validate the initialisers and create a new project or load a existing project.

Argument

  • p: value of type Project

Example

p = Project(
    id="xyz",
    name="My Fancy? *Project1 2 ",
    template="jl",
    profile="default"
)
initiate(p)
source
ProjectFlow.isexistMethod
isexist(m::Array{String}, pid::String)

Internal function to check if a project already exist with the same name with a given profile

Argument

  • m: Array of string with all the path to check.
  • pid: Project id and Project name as string
source
ProjectFlow.load_profilesMethod

Load the profiles from the users profile setup file.

Argument

  • Path of profile file.

Example

const ROOT_DIR = joinpath(homedir(), ".projectflow/profiles")
profiles = load_profiles(ROOT_DIR)
source
ProjectFlow.log_projectMethod

Log a project after consolidation to the given project.

Argument

  • path: path of the log file.
  • prj: The consolidated project dictionary
source
ProjectFlow.pathfinderMethod

Take new porject fullname and profile path. create a new project if the project does not exist or load the path of the project if the project already exist.

Argument

  • meta: Project Metadata after Consolidation
  • prf: loaded profile

return if the project exist as true/false and all relevant path.

source
ProjectFlow.save_templateFunction

make_template(path::String, p::Dict, t::String)

Make a template based on given project info and path and save it to projrct folder.

Argument

  • path: Path to save the cached template
  • p: map of pared values of identifier and replacer
  • t: text templare to replace the identiter with value
source
ProjectFlow.select_propertyMethod

function select_property(path::String, name::String)

Select property of a profile from a given profile value.

Argument

  • path: Path for Profile
  • name: Name of the Profile
Example:
const ROOT_DIR = joinpath(homedir(), ".projectflow/")
select_property(ROOT_DIR, "default")
source
ProjectFlow.set_workflowMethod

function setup_flow(p::Dict)

Set up the initial project flow. Execution of the function will create datalake, insights, vizfiles and datafiles directories in to given project roots.

Argument

  • p: A dictionary auto read by the profiler.
source
ProjectFlow.split_stringMethod

Split profile strings in to property to add in to dict collection.

Argument

  • property string with signature key=pair

Example

split_string("repo=my_beautiful_repo")
source
ProjectFlow.validateMethod

validate(p::Project)

validate a given value of type project.

#Arguemnt

  • p: Value of type project

Example

A_PROJECT = ProjectFlow.Project(
    id="xyz",
    name="My Fancy? *Project1 2 ",
    template="jl",
    profile="pycharm"
)
validate(A_PROJECT)
source
ProjectFlow.write_appendMethod

write_append(path::String, content::String)

Create the file if not exist. If exist write to file.

Argument

  • path: path to write file
  • content: The content to write in the file
source