[SOLVED] comp90024 - Cluster and Cloud  -   Computing  Assignment  2   -   City Analytics  on  the  Cloud        

Description

Background

In            development    and        delivery               of            non-trivial          software             systems,              working                as            part       of            a              team     is             generally             (typically!)        the         norm.   This                assignment        is             very      much    a              group    project. Students              will        be           put         into                software             teams   to            work     on           the         implementation              of            the         system                described            below.                  These    will        be           teams   of            up           to            5             students.                In            this        assignment,       students              need      to            organize              their      team     and        their                collective            involvement     throughout.       There    is             no           team     leader   as            such,     but                teams   may       decide  to            set          up           processes            for          agreeing              on           the         work                and        who       does      what.    Understanding the         dependencies    between              individual                efforts  and        their      successful           integration         is             key        to            the         success of            the                work     and        for          software             engineering        projects               more     generally.            If                teams   have      “issues”,               then       please   let           me         know    asap       and        I              will        help                resolve them.

Assignment

The        software             engineering        activity builds   on           the         lecture materials            describing                Cloud    systems               and        especially           the         UniMelb              Research             Cloud    and        its                use         of            OpenStack;        on           data       from     the         Twitter APIs,     and        CouchDB             and                the         kinds     of            data       analytics             (e.g.       MapReduce)      that        CouchDB             supports                as            well       as            data       from     the         Australian           Urban   Research             Infrastructure                Network             (AURIN                –             https://portal.aurin.org.au).      The        focus     of            this                assignment        is             to            harvest tweets  from     across   the         cities     of            Australia             on                the         UniMelb              Research             Cloud    and        undertake           a              variety of            social                media   data       analytics             scenarios            that        tell         interesting          stories  of            life         in                Australian           cities     and       importantly       how       the         Twitter data       can         be           used                alongside/compared     with/augment  the         data       available             within  the         AURIN  platform                to            improve              our         knowledge         of            life         in            the         cities     of            Australia.                Teams  can         download           data       from     the         AURIN  platform,            e.g.         as            JSON,    CSV                or           Shapefiles,          or           using     the         AURIN  openAPI              (https://aurin.org.au/aurin-apis/).                This       data       can/should        be           included              into        the         team’s  CouchDB             database                for          analysis               with      Twitter data.

The        teams   should  develop               a              Cloud-based      solution               that        exploits                a                multitude           of            virtual  machines            (VMs)   across   the         UniMelb              Research                Cloud    for          harvesting          tweets  through               the         Twitter APIs      (using   both      the                Streaming           and        the         Search  API        interfaces).        The        teams   should  produce               a                solution               that        can         be           run         (in          principle)           across   any        node      of            the                UniMelb              Research             Cloud    to            harvest and        store     tweets  and        scale      up/down                as            required.             Teams  have      been      allocated             4             servers (instances)         with      8                virtual  CPUs     and        500Gb  of            volume storage.                All          students              have      access   to                the         UniMelb              Research             Cloud    as            individual           users     and        can         test/develop                their      applications       using     their      own       (small) VM         instances,            e.g.         using     personal                instances             such      as            pt-1234.              (Remembering that        there     is             no           persistence                in            these     small,    free        and        dynamically      allocated             VMs).

The        solution               should  include a              Twitter harvesting          application         for          any/all of            the                cities     of            Australia.            The        teams   are         expected             to            have      multiple                instances             of            this        application         running                on           the         UniMelb              Research                Cloud    together              with      an           associated          CouchDB             database              containing          the                amalgamated    collection           of            Tweets from     the         harvester            applications.      The                CouchDB             setup     may       be           a              single    node      or           based    on           a              cluster                setup.    The        system should  be           designed              so           that        duplicate             tweets  will        not                arise.

Students              may       want     to            explore other     sources                of            data       they       find        on           the                Internet,              e.g.         information       on           weather,              sport     events, TV          shows,  visiting                celebrities,         stock     market rise/falls,            official statistics              on           Covid-19             however                these     are         not         compulsory       to            complete            the         work.    A             large      corpus  of                Twitter posts     will        be           made    available             for          data       analytics,            but         again                teams   may       decide  that        they       only       wish      to            focus     on           Twitter data       that        they                collect.

Teams  are         expected             to            develop               a              range    of            analytic               scenarios,           e.g. using     the         MapReduce        capabilities        offered by           CouchDB             for          social    media   analytics and        comparing         the         data       with      official data       from     AURIN. Teams  are         free        to explore any        scenarios            that        connect               “in          some     way”     to            the         AURIN  data. Teams  are         encouraged        to            be           creative               here.      A             prize     will        be           awarded for          the         most      interesting         scenarios            identified!

For         example              teams   may       look       at            scenarios            such      as:

• How many    tweets  mention              Covid-19             or           coronavirus       and        are         these clustered             in            certain areas,

e.g.      rich        vs           poor      suburbs               or           in            statistical            areas     where   there     are             more/less           hospitals             etc?

• What do           the         movement         patterns               of            people  look       like        before  Covid-19, during  and        after      lockdown           etc?
• Which suburb has         the         most     tweeters              and        does      this        correlate             with      what we          might    expect  from     the         population         demographic    of            the         suburb from AURIN, g.         more     young   people  live        in            a              given     area       so           we          might expect  a              proportionate   increase               in            the         number               of            tweets  (assuming young   people  tweet    more)?
• Do the         different              languages            used      when    tweeting              correlate             with      the cultures               we          would   expect  to            find        in            those     areas,    g.         more     Chinese live        in            Box        Hill         in            Melbourne         hence    we          would   expect  to            see         for tweets  tagged  as            Chinese                from     those     suburbs,              or           Italians in            Carltonetc?
• Is there     a              correlation         between              crime    related tweets  and        official crime statistics              across   the         suburbs               of            Melbourne?
• Is there     a              correlation         between              alcohol related tweets  or           crime    and locations             of            places   to            buy        alcohol (bottleshops)?
• Does language              use,        g.         vulgar   words   used      in            Twitter happen more     or           less in            wealthy               or           poor      areas?

The        above   are         examples            –             students              may       decide  to            create   their      own                analytics             based    on           the         data       they       obtain.  Students              are         not         expected                to            build     advanced            “general               purpose”             data       analytic               services               that                can         support                any        scenario              but         show     how       tools      like        CouchDB             with                targeted               data       analysis               capabilities        like        MapReduce        when    provided             with                suitable                inputs   can         be           used      to            capture                the         essence                of            life                in            Australia.

A             front-end            web       application         is             required              for          visualising          these     data                sets/scenarios.

For         the         implementation,             teams   are         recommended to            use         a              commonly                understood        language              across   team     members            –             most     likely    Java       or                Python. Information      on           building               and        using     Twitter harvesters          can         be                found    on           the         web,      e.g.         see         https://dev.twitter.com/           and        related links      to                resources            such      as            Tweepy               and        Twitter4J.           Teams  are         free        to            use                any        pre-existing       software             systems               that        they       deem    appropriate       for          the                analysis               and        visualisation      capabilities,       e.g.         Javascript           libraries,              Googlemaps                etc.

Error   Handling

Issues   and        challenges          in            using     the         UniMelb              Research             Cloud    for          this                assignment        should  be           documented.     You        should  describe              the         limitations         of                mining twitter content and        language              processing          (e.g.       sarcasm).            You        should                outline any        solutions             developed          to            tackle   such      scenarios.           The        database                may       however             contain re-tweets.           You        should  demonstrate     how       you        tackled                working              within  the         quota    imposed              by           the         Twitter APIs      through               the                use         of            the         Cloud.