ProjectFlow
ProjectFlow is a ad-hoc analytics project manager. Upon creating a new type Project
and providing it to initiate
function will crate all necessary directory into local machine to start working on the project.
Steps:
- First we need to create a profile which ProjectFlow will use to set up initial
directories. Profile should be stored in /home/username/.projectflow/
directory. inside of .projectflow
we will create a file called profiles
and add the profile name and properties. Inside of priofile file should looks like this:
[default]
project_root=/home/datapsycho/JuliaProjects/adhocs
projects_dir=projects
data_dir=datalake
insights_dir=insights
insights_viz_dir=vizfiles
insights_data_dir=datafiles
The convention must be followed and all these directory map should be there. Here it say our project root where all the projects will be created. Our raw data will be stored in datalake
our visuals will be stored in insights/vizfiles
and our post analysis report data will be stored in insights/datafiles
.
The Root Project folders must be created before going to next step. which is JuliaProjects/adhocs
in the examples.
- Next we have to create type called project and run the
Initiate
method.
Either it will crate a new project or it will load the paths of existing project.
using ProjectFlow
p = ProjectFlow.Project(
id="xyz",
name="My Fancy? *Project1 2 ",
template="jl",
profile="default"
)
datalake, iviz, idata = initiate(p)
The following execution will create the directories in adhocs
directory such as:
JuliaProjects/adhocs/
├── datalake
│ └── 2020-07-18_My_Fancy_Project_xyz
├── insights
│ └── 2020-07-18_My_Fancy_Project_xyz
│ ├── datafiles
│ └── vizfiles
└── projects
└── 2020-07-18_My_Fancy_Project_xyz
└── code.jl
- Inside of the code.jl there will already boiler plate code to start project.
# Project Name:
# Regular Imports
using ProjectFlow
p = Project(
id="xyz",
name="My Fancy? *Project1 2 ",
template="jl",
profile="default"
)
datalake, idata, iviz = initiate(p)
Which is the starting point of start writing adhoc dataanalytics single file project.
You might want to create a separate environment for whole adhocs project and an initializer jl file filled up with init code and use it every time before starting a new project.
# initializer.jl file
using ProjectFlow
p = ProjectFlow.Project(
id="xyz",
name="My Fancy? *Project1 2 ",
template="jl",
profile="default"
)
datalake, iviz, idata = initiate(p)
Internal Methods Index
ProjectFlow.Project
ProjectFlow.cache_template
ProjectFlow.clean_name
ProjectFlow.consolidate
ProjectFlow.initiate
ProjectFlow.isexist
ProjectFlow.load_profiles
ProjectFlow.log_project
ProjectFlow.pathfinder
ProjectFlow.profiles_exist
ProjectFlow.save_template
ProjectFlow.select_property
ProjectFlow.set_workflow
ProjectFlow.split_string
ProjectFlow.validate
ProjectFlow.validate_profiles
ProjectFlow.write_append
ProjectFlow.Project
— TypeImmutable type Project. Project hold:
- Unique Project Id
- Project name
- Template extention jl tocreate .jl file, in future will have option for .ipynb notebook
- The profile to use when creating projects, the default value is
default
ProjectFlow.cache_template
— Methodcache_template(p::Dict, t::String)
Load the template in the memory and replace the idetifier with given value of stirng.
Argument
- p: map of pared values of identifier and replacer
- t: text templare to replace the identiter with value
ProjectFlow.clean_name
— Methodfunction clean_name(s::String)
Take a strign remove spacial character from the string
Argument
- s: String to remove special character
Example:
clean_name("Here is < 12 ?")
ProjectFlow.consolidate
— Methodconsolidate(p::Project)
Consolidate a value of type project and return a Dict object with cleaned project name and newly added attribute.
#Arguemnt
- p: Value of type project
Example
A_PROJECT = ProjectFlow.Project(
id="xyz",
name="My Fancy? *Project1 2 ",
template="jl",
profile="pycharm"
)
consolidate(A_PROJECT)
ProjectFlow.initiate
— Methodfunction initiate(p::Project)
This is the main function compose all other function together and upon giving value of type project function will validate the initialisers and create a new project or load a existing project.
Argument
- p: value of type Project
Example
p = Project(
id="xyz",
name="My Fancy? *Project1 2 ",
template="jl",
profile="default"
)
initiate(p)
ProjectFlow.isexist
— Methodisexist(m::Array{String}, pid::String)
Internal function to check if a project already exist with the same name with a given profile
Argument
- m: Array of string with all the path to check.
- pid: Project id and Project name as string
ProjectFlow.load_profiles
— MethodLoad the profiles from the users profile setup file.
Argument
- Path of profile file.
Example
const ROOT_DIR = joinpath(homedir(), ".projectflow/profiles")
profiles = load_profiles(ROOT_DIR)
ProjectFlow.log_project
— MethodLog a project after consolidation to the given project.
Argument
- path: path of the log file.
- prj: The consolidated project dictionary
ProjectFlow.pathfinder
— MethodTake new porject fullname and profile path. create a new project if the project does not exist or load the path of the project if the project already exist.
Argument
- meta: Project Metadata after Consolidation
- prf: loaded profile
return if the project exist as true/false and all relevant path.
ProjectFlow.profiles_exist
— MethodInternal profile checker which check if the config file exist or not.
Argument
path
: File path of the profiles
ProjectFlow.save_template
— Functionmake_template(path::String, p::Dict, t::String)
Make a template based on given project info and path and save it to projrct folder.
Argument
- path: Path to save the cached template
- p: map of pared values of identifier and replacer
- t: text templare to replace the identiter with value
ProjectFlow.select_property
— Methodfunction select_property(path::String, name::String)
Select property of a profile from a given profile value.
Argument
- path: Path for Profile
- name: Name of the Profile
Example:
const ROOT_DIR = joinpath(homedir(), ".projectflow/")
select_property(ROOT_DIR, "default")
ProjectFlow.set_workflow
— Methodfunction setup_flow(p::Dict)
Set up the initial project flow. Execution of the function will create datalake, insights, vizfiles and datafiles directories in to given project roots.
Argument
- p: A dictionary auto read by the profiler.
ProjectFlow.split_string
— MethodSplit profile strings in to property to add in to dict collection.
Argument
- property string with signature key=pair
Example
split_string("repo=my_beautiful_repo")
ProjectFlow.validate
— Methodvalidate(p::Project)
validate a given value of type project.
#Arguemnt
- p: Value of type project
Example
A_PROJECT = ProjectFlow.Project(
id="xyz",
name="My Fancy? *Project1 2 ",
template="jl",
profile="pycharm"
)
validate(A_PROJECT)
ProjectFlow.validate_profiles
— MethodInvalidate the loaded properties from profiles config file.
Argument
p
profiles Dict will all profiles from config file
ProjectFlow.write_append
— Methodwrite_append(path::String, content::String)
Create the file if not exist. If exist write to file.
Argument
- path: path to write file
- content: The content to write in the file