[SOLVED] CSE510-Project 1 Data Analysis

Description

Our first project is focused on preprocessing, analysing and visualizing real-world datasets. Applying the basic statistical methods and extracting valuable summary about it. During the third part of the project you are expected to employ multiple datasets and extract insights from them.

Dataset

Requirements to the dataset for Part I and II:

• Represent the real-world data
• Contain at least 50k entries
• Should be different from the one used in class

Possible resources includes:

• Open Data Buffalo – https://data.buffalony.gov​ /
• Google Dataset – https://datasetsearch.research.google.com​ /
• US Government’s Data https://www.data.gov​ /

Tasks

Part I: Perform data analysis of the dataset

1. How many entries and variables does the data set comprise?
2. What types of data is included?
3. Are there any data missing?
4. Provide the main statistics about the entries of the dataset (mean, std, etc.)
5. Visualize the data (min 3 graphs), e.g. correlation between different variables. Are there any interesting patterns?

Part II: Apply ML analysis

1. Choose the features and targets in the dataset.
2. Preprocess the dataset for training (e.g. cleaning and filling the missing variables, split between training/testing/validation).
3. Apply ML algorithms (min 3 algorithms) to model the target variable. This can be either classification or regression task. You can use any of the libraries with inbuilt ML functions.
4. Provide the comparison of the results of different ML models you have used. This can be in the form of graph representation and your reasoning about the results.

Part III: Employ multiple datasets and extract insights

1. Choose any related dataset to your current one. Combine the two into one dataset. The combined dataset doesn’t have size requirements.
2. Choose the correlated variables.
3. Perform statistical analysis on finding the correlation between selected features from both datasets. Examples:
   1. Find the correlation between the crime and the number of schools in the area.
   2. Find the correlation between the traffic and the population in the area
4. Analyse the results and any interesting patterns.