MacPorts webapp project updates

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

MacPorts webapp project updates

Arjun Salyan
Dear all,

This email is regarding updates and future plans of the webapp (ports.macports.org) enhancement project that I am working on as a part of GSoC 2020 under the mentorship of Mojca, Amar Takhar, and Rajdeep Bharati.

We started with modifying the codebase to make it simpler but solid. This would also make things easier for contributors. The project has been split into smaller apps, and refactoring is still underway, being done app-by-app.

A new advanced search feature has been added (demo links below, under testing as of now) using Solr search backend. Adding Solr as a dependency would require more resources, but unfortunately, the current server is already being utilized to its capacity. We, therefore, need to look for other options to host the webapp, one of which is acquiring a VM from OSU Open Source Lab (suggested by Amar). We also plan to add a caching engine so that exploring the webapp remains smooth even with the ever-growing port statistics data. This makes the need for a new host even more important.

The API has been completed (except for port stats, which is under construction), everything stored by the webapp would be accessible easily for other implementations. I am writing an OpenAPI spec for usage instructions.

This week we will finish adding Livecheck results on the webapp, which is being achieved by having a working copy of MacPorts in an ubuntu based docker container. This would allow filtering and viewing the ports which have a newer version of the software available. Another feature to be added is the list of the files installed by a port (will be fetched from buildbot).

I will keep posting updates about future plans and completed tasks. Any feature requests, feedback, or suggestion would be very constructive. 

P.S: The changes have not been merged to the master and can only be tried on the demo links given below.

Ubuntu image (running macports) : https://hub.docker.com/r/arjunsalyan/macports-ubuntu
Reply | Threaded
Open this post in threaded view
|

Re: MacPorts webapp project updates

ryandesign2
Administrator


On May 17, 2020, at 11:48, Arjun Salyan wrote:

> We also plan to add a caching engine so that exploring the webapp remains smooth even with the ever-growing port statistics data. This makes the need for a new host even more important.

Not sure specifically what you're planning with regard to caching, but note that we already have an arrangement with the MaxCDN content delivery network and we currently use it for www, guide, distfiles and packages. We can also use it for the ports webapp if desired. It would just be a matter of tuning the settings to decide how long the data should be cached.
Reply | Threaded
Open this post in threaded view
|

Re: MacPorts webapp project updates

Mojca Miklavec-2
Dear Ryan,

On Mon, 18 May 2020 at 21:47, Ryan Schmidt wrote:
> On May 17, 2020, at 11:48, Arjun Salyan wrote:
>
> > We also plan to add a caching engine so that exploring the webapp remains smooth even with the ever-growing port statistics data. This makes the need for a new host even more important.
>
> Not sure specifically what you're planning with regard to caching, but note that we already have an arrangement with the MaxCDN content delivery network and we currently use it for www, guide, distfiles and packages. We can also use it for the ports webapp if desired. It would just be a matter of tuning the settings to decide how long the data should be cached.

The caching is primarily not so much about big data files (or actual
troubles with tons of visitors) as it is about whether the query "Show
me on which macOS versions the wine port is broken" needs to be
executed in the database every single time any random visitor opens or
refreshes the port page (even if the port didn't see any changes for
months and meanwhile thousands of visitors query the page). I don't
know to what extent MaxCDN can help here given that the main problem
is the lack of memory required to even run the site on the existing
server.

We can certainly check whether we could make use of MaxCDN (I don't
know to what extent a page that's dynamic in nature can be supported
there), but the immediate "urgency" is to find an alternative hosting
solution altogether.

Mojca
Reply | Threaded
Open this post in threaded view
|

Re: MacPorts webapp project updates

ryandesign2
Administrator


On May 18, 2020, at 16:29, Mojca Miklavec wrote:

> On Mon, 18 May 2020 at 21:47, Ryan Schmidt wrote:
>> On May 17, 2020, at 11:48, Arjun Salyan wrote:
>>
>>> We also plan to add a caching engine so that exploring the webapp remains smooth even with the ever-growing port statistics data. This makes the need for a new host even more important.
>>
>> Not sure specifically what you're planning with regard to caching, but note that we already have an arrangement with the MaxCDN content delivery network and we currently use it for www, guide, distfiles and packages. We can also use it for the ports webapp if desired. It would just be a matter of tuning the settings to decide how long the data should be cached.
>
> The caching is primarily not so much about big data files (or actual
> troubles with tons of visitors) as it is about whether the query "Show
> me on which macOS versions the wine port is broken" needs to be
> executed in the database every single time any random visitor opens or
> refreshes the port page (even if the port didn't see any changes for
> months and meanwhile thousands of visitors query the page). I don't
> know to what extent MaxCDN can help here given that the main problem
> is the lack of memory required to even run the site on the existing
> server.
>
> We can certainly check whether we could make use of MaxCDN (I don't
> know to what extent a page that's dynamic in nature can be supported
> there), but the immediate "urgency" is to find an alternative hosting
> solution altogether.

The CDN could probably help. If we set that up for the web app, then when a user requests a port page the CDN would check if it has a cached copy of the page and it is not "too old", and if so it prints the cached copy. It only contacts the origin server if it doesn't already have an up-to-date version of the page.

The origin server gets to decide what "too old" means by setting the Cache-Control headers that specify when the contents of the page should be considered expired. Or the origin server's headers can be overridden by settings in the CDN. How to set the page expiration time depends on how old you want content to be able to get. For example, if I update a port today, what's the maximum amount of time you want users to have to wait before they see that change reflected on the web? Or tickets or build results or whatever other information is part of the page.

Reply | Threaded
Open this post in threaded view
|

Re: MacPorts webapp project updates

Arjun Salyan
Dear all,

This is in continuation to my previous email regarding the ongoing up-gradation of the webapp.

Finished tasks
- Using a local macports installation (docker) we are able to display livecheck results on the webapp.

- The advanced search page is almost complete in functionality and now provides the ability to filter and find exactly what a user might need. Livecheck filters can also be applied to find outdated ports or the ports with broken livecheck. Example: http://macports.silentfox.tech/search/?livecheck_outdated=on
Filters for maintainers, categories and variants can also be applied. The filters support autocompleting and suggestions.

- Files installed by a port are now stored in the database alongside build history and displayed with port health. Example: http://macports.silentfox.tech/port/p5.30-dist-zilla/

- The port page has been updated keeping in mind previous suggestions and open issues. The code has also been simplified for easier maintenance.

- A twitter typeahead based search bar has been added to the hompage and the header of all pages. It helps in quickly finding ports by matching with the names, it also supports autocompleting.

Ongoing tasks:
- Improve page load time for statistics by drawing the charts after asynchronously fetching data for each. This is being done on top of the improved queries and a plan of caching the results (options: use of CDN as suggested by Ryan, utilising django's support for in-memory caching). http://macports.silentfox.tech/port/wget/stats/

- Drop dependency on Google Charts

- Finish the API endpoints for stats

 - Allow searching for ports by the files installed. We could really use some suggestions on this, as to what type of lookups (exact, contains, startswith, endswith etc.) would be good while allowing to search ports by the files they install. Also, shall we consider matching with the full path, just the file, or both?

Any feature requests or feedback would be helpful.

Thank you